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L»t.  Col.  Roy  M.  Gulick 
Advanced  Research  Projects  Agency 
Cybernetics  Technology  Office 
®  1400  Wilson  Boulevard 

Arlington,  Virginia  22209 

Dear  Col.  Gulick: 

•  With  this  letter,  I  am  pleased  to  transmit  the  Final  Report  of  our 

contract  for  the  "Development  and  Evaluation  of  a  Bayesian  Sequential 
Testing  Methodology  for  Assessing  the  Reliability  of  Defense  Systems.  " 


•  A 


This  Report  is  organized  into  five  parts: 

PART  1  provides  an  introduction  and  summary  of  the 
Project,  reviewing  its  objectives  and  av-complishments. 


PART  2  describes  one  example  of  the  theoretical  savings 
that  can  be  achieved  for  a  simple  system  when  a  cost- 
#  effective  test  plan  is  adopted. 


PART  3  is  a  User's  Manual  for  the  FORTRAN  IV  computer 
program  developed  in  this  project  for  carrying  out  Bayesian 
reliability  assessments  and  for  implementing  the  sequential 
testing  methodology. 


PART  4  presents  a  directory  of  organizations  in  the 
Department  of  Defense  involved  in  reliability  assessment 
and  the  results  of  a  survey  on  their  attitudes  to  Bayesian 
reliability  assessment. 


PART  5  provides  a  copy  of  the  briefing  charts  used  to 
explain  the  project  to  various  members  of  the  Armed 
Se  rvices. 


We  have  thoroughly  enjoyed  working  on  this  contract  with  you  and 
Dr.  Clinton  Kelly  of  Decisions  and  Designs,  Inc.  Possibly,  we  have  made 
a  worthwhile  contribution  to  Bayesian  reliability  assessment  and  the  concept 
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of  cost-effective  testing  in  general.  We  hope  to  continue  our  efforts  in 
this  field  and  look  forward  to  the  opportunity  of  making  future  contribur 
tions  to  the  Department  of  Defense. 

Any  comments  you  may  have  cn  the  report  will  be  most  welcome. 

Yours  truly, 

David  V.  Mastran 
President 
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PART  1 

INTRODUCTION  AND  SUMMARY 


In  the  first  part  of  the  Final  Report  we  review  the  concept  of  cost- 
effective  testing,  the  basic  objective  of  this  Project,  the  task  plan  followed, 
and  the  accomplishments  at  completion. 


I.  CONCEPT  OF  COST-EFFECTIVE  TESTING 

y^The  concept  of  cost-effective  testing  of  multi -component  systems  is 
central  to  this  Project.  The  concept  arose  from  a  study  of  the  testing  of 
operational  systems  aimed  at  detecting  the  degradation  of  component  relia¬ 
bilities  over  time.  This  type  of  testing  is  usually  conducted  on  a  continual 
basis  by  system  users  to  guard  against  the  erosion  of  system  reliability 
due  to  aging.  Typically,  the  system,  consisting  of  various  subsystems 
or  components,  is  brought  in  from  the  field  for  testing.  The  components 
are  then  tested  in  two  ways:  either  independently  in  component- level  tests, 
or  simultaneously  in  system  associated  tests.  —  j  <-,/]  -3  - 

As  these  systems  are  brought  to  the  test  facility,  each  component 
in  the  system  is  tested.  As  a  result,  the  total  number  of  tests  on  each 
component  is  a  direct  function  of  the  number  of  times  it  appears  in  the 
system.  Testing  all  the  components  in  the  system  does  not  consider  the 
variations  in  both  the  cost  and  value  of  the  information  obtained  from 
testing  each  component.  These  variations  can  be  important  in  developing  I 
a  cost-effective  test  plan,  especially  when  one  is  faced  with  a  limited 


test  budget. 
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The  concept  of  cost-effective  testing  also  extends  to  other  types  of 
testing,  including,  for  example,  production-line  testing.  As  items  are 
manufactured  in  production  lines  and  aggregated  into  lots,  they  are  tested 
by  the  contractor  and/or  government  personnel  for  the  purpose  of  accept¬ 
ing  or  rejecting  the  lot.  These  components  are  then  combined  into  larger 
and  larger  assemblies  which  are  also  aggregated  into  lots  and  tested.  The 
largest  assemblies  are  combined  into  subsystems  and  are  tested  again. 
Areas  that  offer  possible  reduction  in  testing  costs  are  avoiding  duplicate 
testing  and  designing  lot  sizes  to  consider  the  serial  correlation  in  lot 
test  results.  Thus,  the  process  of  testing  multicomponent  systems  cost- 
effectively  is  a  much  more  difficult  problem  than  testing  components 
independently . 

The  savings  achievable  from  cost-effective  testing  can  occur  in  a 
number  of  different  ways.  Under  certain  circumstances,  it  may  be 
advantageous  to  test  some  components  in  the  system  much  more  than 
others,  or  not  to  test  some  components  at  all.  For  example,  some  com¬ 
ponents  might  be  more  costly  to  test  than  others;  in  destructive  testing, 
the  component  has  to  be  replaced.  In  other  cases,  special  test  equip¬ 
ment  or  procedures  are  required,  or  components  are  inaccessible, 
and,  therefore,  costly  to  remove  or  monitor.  In  still  other  cases, 
more  information  (both  objective  and  subjective)  might  be  available 
on  some  components  than  on  others.  (As  systems  evolve  other  time, 
components  from  the  preceding  generation  of  the  system  often  are  used 
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in  the  current  generation;  consequently,  past  test  data  and  engineering 
judgment  are  available  and  can  be  used  to  reduce  testing  requirements.  ) 
Or,  because  of  the  component's  position  in  the  system  and  number  of 
redundancies,  some  components  may  b-  more  critical  to  the  successful 
functioning  of  the  entire  system  and,  t.ence,  should  be  tested  more  than 
others.  Components  in  series,  obviously,  are  the  most  critical. 


C*nOf 

- In  sum,  there  are  a  number  of  reasons  why  it  is  possible  that  all 

components  in  the  system  should  not  be  tested  the  same  or  a  propor¬ 
tionate  number  of  times.  The  purpose  of  this  project  is  to  evaluate 
a  Bayesian  sequential  testing  methodology  that  considers  these  reasons 
and  balances  the  costs  of  testing  with  the  expected  value  of  the  informa¬ 
tion  to  be  gained.  The  methodology  indicates  in  sequential  form  which 
component  or  subsystem  in  the  system  to  test  next,  and  when  to  stop 
testing.  This  test  plan  developed  should  provide  estimates  of  the 
reliability  of  complex  systems  more  cost-effectively  than  the  plans 
typically  used. 

The  methodology  examined  in  this  project  uses  a  sequential  testing 
scheme  for  obtaining  attribute  (pass-fail)  test  data  from  components  of 
multi- component  systems.  The  components  in  the  system  can  be  con¬ 
figured  in  any  manner  whatsoever,  assuming  that  the  reliability  of  the 
system  can  be  expressed  as  a  function  of  the  reliabilities  of  the  com¬ 
ponents.  The  selection  of  the  prior  distribution  of  component  reliabil¬ 


ities  and  the  form  of  the  loss  function  are  also  completely  flexible. 
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The  objective  of  the  cost-effective  testing  methodology  is  to 
present  decision  rules  for  minimizing  the  decision  maker's  expected 
loss  in  terms  of  future  risk  and  the  component  testing  costs.  Bayesian 
preposterior  analysis  is  used  to  determine  when  the  cost  of  testing  a 
component  exceeds  the  sayings  that  can  be  expected  to  result  from  re¬ 
ducing  risk.  Risk,  here,  is  defined  as  the  expected  loss  due  to  mis- 
estimation. 

The  test  plan  is  constructed  as  follows.  Before  a  component  is 
tested,  a  calculation  is  made  to  determine  if  the  combination  of  the 
future  risk  and  the  cost  of  testing  can  be  expected  statistically  to  de¬ 
crease.  The  current  value  of  the  risk  is  known  as  the  prior  risk,  even 
though  some  of  the  components  might  have  already  been  tested.  The 
posterior  risk  is  simply  the  new  risk  after  testing.  The  prior  expecta¬ 
tion  of  the  posterior  risk  is  the  expected  value  of  the  risk  before  the 
additional  tests  are  made. 

The  basic  decision  is  when  to  stop  testing,  and  if  not,  which  com¬ 
ponent  to  test  next.  The  decision  process  is  sequential,  test  by  test, 
and  determines  in  a  series  of  decision  rules  which  component  offers 
the  greatest  expected  reduction  in  total  loss,  given  the  series  of 
previous  test  results. 

II.  OBJECTIVE  OF  THE  PROJECT  AND  TASK  PLAN 

The  basic  objective  of  this  project  was  to  evaluate  the  cost  savings 
potential  of  a  testing  plan  developed  for  multi -component  systems. 
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considering  the  cost  and  value  of  the  information  obtained  from  testing 
selected  components.  The  methodology,  which  was  partially  developed 
at  the  time  the  project  began,  promised  to  result  in  significant  savings  in 
the  cost  of  testing  and  greater  precision  in  the  estimates  obtained  from 
testing.  Four  basic  tasks  were  undertaken. 

Task  1:  Review  Current  Service  Testing  Programs  and  Refine  Methodology 


In  this  task,  the  various  testing  programs  of  the  Army,  Navy,  and 
Air  Force  were  surveyed  concerning  the  nature  and  structure  of  their 
testing  programs  and  their  attitudes  toward  Bayesian  analyses.  We  wished 
first  to  insure  that  the  Service  program  selected  for  this  project  was  recep¬ 
tive  to  demonstrating  the  advantages  of  the  methodology.  Second,  we  desired 
to  gain  a  greater  understanding  of  the  variety  of  organizations  in  each  ser¬ 
vice  involved  in  reliability  assessment.  The  results  of  the  survey  and  a 
directory  of  these  organizations  are  presented  in  Part  4  of  this  report. 

Task  2:  Develop  Computer  Programs  for  Implementing  Methodology 


Experience  has  shown  that  a  sophisticated  decision  analysis  method¬ 
ology  has  a  better  chance  of  implementation  if  it  is  supported  by  computer 
software.  In  this  task,  we  developed  a  computer  program  which  would 
facilitate  the  development  of  a  cost-effective  test  plan.  The  program 
named  ABRAM  (Automated  Bayesian  Reliability  Assessment  Model)  is 
described  in  Part  3  of  this  report. 
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Task  3:  Select  a  Testing  Organization  and  Instruct  the  Staff  in  the  Applica- 


tion  of  the  Methodology 

In  this  task,  the  Army  Armament  Materiel  Readiness  Command  was 
selected  to  participate  in  the  project  and  implement  the  methodology  on  an 
armament  system.  A  trip  was  made  to  Rock  Island  Arsenal,  Illinois,  in 
October,  1977  to  explain  the  computer  program  and  brief  key  officials  on 
the  project.  Mr.  Louis  Iannuzzelli  was  designated  the  ARRCOM  Project 
Manager,  and  Mr.  Robert  McKeague  a  project  participant.  A  description 
of  ARRCOM' s  mission  and  organizational  structure  and  the  actual  system 
selected  for  test  are  provided  in  the  briefing  charts  in  Part  5  of  the  report. 


Task  4:  Evaluate  Organizations'  Experience  with  the  Methodolo 


This  task  was  to  result  in  a  definitive  statement  of  the  cost  savings 
potential  of  the  methodology  at  ARRCOM.  Unfortunately,  no  data  were 
available  on  the  cost  of  testing  components  and  considerable  effort  would 
have  been  required  to  collect  the  data.  Because  of  the  limited  contract 
funds  and  time  remaining,  and  the  different  type  of  testing  being  conducted 
(production-line  vs.  surveillance  for  degradation),  it  was  decided  that  a 
theoretical  study  using  simulated  data  would  have  to  be  substituted.  Part 
2  of  this  report  describes  this  study  and  shows  that,  in  fact,  substantial 
savings  are  possible. 

In  sum,  although  the  project  did  not  demonstrate  on  a  real  system  the 
level  of  cost  savings  possible,  it  did  show  that  such  a  demonstration  would 
probably  be  very  successful.  It  also  produced  a  great  deal  of  background 


material  that  should  be  helpful  as  independent  products. 
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in.  SPECIFIC  ACCOMPLISHMENTS  OF  THE  PROJECT 

The  specific  accomplishments  of  the  project  were  as  follows. 

First,  a  survey  was  conducted  of  the  receptivity  of  major  testing 
programs  in  all  the  military  services  to  Bayesian  reliability  assessment 
techniques.  Persons  throughout  the  DoD  testing  community  were  briefed 
on  the  project,  including  the  Reliability  Implementation  Group  (RIG)  in 
the  Air  Force.  Considerable  interest  was  expressed  in  the  basic  con¬ 
cept  and  further  study  was  recommended. 

The  organizational  structure  and  composition  of  the  DoD  testing 
community  was  studied  and  described  for  purposes  of  promoting  greater 
cooperation  and  understanding  of  the  problems  of  reliability  assessment. 
The  directory  produced  should  also  prove  valuable  for  disseminating 
research  findings  throughout  DoD. 

Second,  the  U.S.  Army  Armament  Materiel  Readiness  Command 
(ARRCOM),  the  organization  responsible  for  the  logistical  support  of 
armament  systems,  selected  to  participate  in  the  study,  was  briefed 
on  the  features  of  the  methodology.  ARRCOM,  recognizing  that  the 
methodology  could,  in  fact,  result  in  substantial  savings  in  test  costs, 
initiated  a  new  effort,  Optimum  Test  at  Minimum  Cost  (OTMC),  to 
assess  the  advantages  of  cost-effective  testing.  A  multi-component 
system  (the  81mm  HE  cartridge  M374)  was  selected  for  analysis  by 
ARRCOM  after  consideration  of  several  alternatives,  and  a  data 
collection  plan  was  developed  to  obtain  information  needed  to  imple- 
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ment  the  methodology  for  proving  grounds  testing. 

ARRCOM  became  actively  interested  in  trying  additional  cost-reduction 
techniques,  including  methods  for  combining  system  and  component  level 
data  in  the  same  assessment,  and  has  expressed  a  desire  to  expand  the  study 
to  other  armament  systems. 

Third,  a  computer  software  package,  Automated  Bayesian  Reliability 
Assessment  Model  (ABRAM),  for  implementing  the  sequental  testing  meth¬ 
odology  was  written  and  a  User's  Manual  was  prepared.  This  program, 
structured  for  use  by  non-technical  users,  will  facilitate  Bayesian  reliability 
assessments  for  interested  organizations. 

Finally,  a  theoretical  study  was  conducted  of  the  potential  savings  that 
can  be  achieved  by  testing  cost-effectively.  A  simple  series /parallel  system 
was  used  to  provide  insight  into  when  significant  savings  can  be  expected. 
Under  fairly  conservative  assumptions,  a  reduction  of  61%  of  the  test  budget 
was  found  possible,  suggesting  promising  results  for  applications  on  real 


systems. 
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PART  2 


DEMONSTRATION  OF  SAVINGS  THAT  MAY  BE 
FEASIBLE  BY  TESTING  COST  EFFECTIVELY 


I.  INTRODUCTION  AND  BACKGROUND 


The  savings  that  may  be  achieved  by  adopting  a  cost-effective  test 
plan  are  demonstrated  in  this  part  of  the  report.  Since  actual  cost 
data  were  not  available  in  time,  simulated  data  have  been  used  to 
demonstrate  the  theoretical  savings  that  are  possible.  Moreover, 
because  of  limited  contract  funds,  only  a  simple  system  was  evaluated. 
Hopefully,  the  results  will  provide  insight  into  the  savings  that  can  be 
obtained  in  more  complex  systems* 

Although  a  complete  description  of  ABRAM,  the  software  program 
used  to  compute  the  savings,  is  provided  in  Part  3,  a  brief  review  of 
the  assumptions  implicit  in  the  program  is  given  here. 

The  main  components  of  any  Bayesian  analysis  are  the  specifica¬ 
tion  of  the  prior  distributions  of  reliability,  the  loss  function  measuring 
the  loss  incurred  by  misestimating  the  true  reliability,  and  the  com¬ 
ponent  and  system  test  results.  Because  ABRAM  is  based  on  the 
assumption  of  squared  error  loss  function  (the  most  common  form 
used  in  reliability  estimation),  only  the  first  two  moments  of  the  prior 
distribution  of  reliability  need  be  specified.  That  is,  a  complete 
Bayesian  analysis  is  available  from  the  moments  of  the  reliability 


distribution  because: 
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•  the  Bayesian  estimate  of  reliability  is  the  mean  (first  moment) 
of  the  distribution;  and 

•  the  expected  loss  (risk)  is  proportional  to  the  variance 
(second  moment  minus  first  moment  squared)  of  the  dis¬ 
tribution. 

For  simplicity  and  clarity,  we  have  selected  the  beta  family  of 
distributions  to  serve  as  the  parametric  form  of  the  prior  distribution. 
Actually,  since  only  the  first  two  moments  of  the  beta  distribution  are 
utilized,  the  selection  of  the  beta  was  made  principally  to  allow  the  user 
of  ABRAM  to  specify  his  degree  of  prior  belief  in  terms  of  a  well-known 
family.  The  use  of  the  beta  parameters,  pseudo-successes  and  pseudo¬ 
failures,  should  facilitate  the  specification  of  the  first  two  moments 
on  the  part  of  the  user,  since  (as  shown  in  Part  3)  these  parameters 
are  readily  interpretable. 

The  test  results  that  can  be  used  by  ABRAM  may  be  either  the 
complete  system,  the  subsystem,  and/or  individual  components.  When 
test  results  are  entered  at  other  than  the  component  level,  the  validity 
of  the  use  of  the  entire  beta  distribution  as  the  form  of  the  updated 
prior  distribution,  rather  than  its  first  two  moments,  is  in  question. 
Nonetheless,  in  practice,  the  error  introduced  by  fitting  a  beta  distri¬ 
bution  appears  to  be  very  small  and  well  worth  the  convenience. 

The  basic  objective  of  the  analysis  is  to  identify  that  component 
which  reduces  the  "prior  expectation  of  the  posterior  risk"  of  the  system 
reliability  at  least  cost.  This  value  is  determined  by  calculating  both 
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the  system  risk  that  would  occur  if  the  component  to  be  tested  resulted 
in  a  success,  and  the  system  risk  that  would  occur  if  the  component 
to  be  tested  resulted  in  a  failure.  These  two  values  of  risk  are  then 
weighted  by  the  current  estimate  of  the  reliability  and  unreliability 
of  the  component  in  question  and  summed.  This  sum,  representing 
an  expected  risk,  is  subtracted  from  the  current  value  of  the  system 
risk  and  divided  into  the  cost  of  testing  the  component.  The  resulting 
value  is  the  cost  of  achieving  a  unit  reduction  in  risk.  The  component 
which  reduces  the  risk  at  least  cost  is  then  selected  for  the  next  test. 

n.  FRAMEWORK  FOR  THE  COMPARISON  METHODOLOGY 

As  mentioned  earlier,  no  cost  data  from  test  programs  were 
available.  Therefore,  we  decided  to  employ  a  predetermined  set  of 
simulated  test  results  to  identify  the  savings  that  could  be  achieved 
by  a  cost-effective  test  plan.  The  basic  approach  was  to  select  a 
hypothetical  system  and  test  it  in  a  conventional  manner  until  a  pre- 
established  test  budget  was  exhausted.  Then,  the  same  set  of  test 
data  would  be  called  upon  in  a  sequential  test  plan  until  the  system  risk 
was  reduced  to  the  level  achieved  by  the  conventional  test  plan.  The 
cost  to  achieve  this  level  of  risk  would  then  be  compared  to  the  original 
test  budget  to  see  what  savings,  if  any,  were  achieved. 

The  conventional  test  plan  was  structured  so  that  each  component  in 
the  system  would  be  tested  a  proportionate  number  of  times;  that  is. 
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each  component  in  the  system  would  be  tested  even  though  it  appeared 
more  than  once  in  the  system.  For  example,  for  two  identical  com¬ 
ponents  connected  in  parallel,  twenty  tests  would  be  conducted  on  the 
component  in  a  system  tested  ten  times.  This  type  of  testing,  as  noted 
earlier,  is  commonly  associated  with  surveillance  testing  for  detecting 
degradation  of  component  reliabilities. 

The  simulated  test  data  constructed  for  each  component  were 
designed  to  reflect  the  system  test  data  used  in  the  conventional  test  plan. 
This  insured  that  the  final  estimates  of  system  reliability  were  not  widely 
divergent  because  of  an  unexpected  failure  occurring  early  or  late  in 
the  sequence.  Moreover,  both  test  plans  started  with  the  same  prior 
distribution  of  system  reliability  to  insure  an  equitable  point  of  de- 
pa  rture. 

A  question  not  addressed  in  the  analysis,  however,  was  how  the 
estimate  of  system  reliability  resulting  from  both  the  conventional 
and  cost  effective  test  plans  compares  to  the  actual  reliability  as 
testing  progresses.  Although  it  can  be  shown  that  the  cost-effective 
test  plan  converges  to  the  true  reliability  as  the  number  of  tests 
increases,  it  is  not  clear  how  the  estimate  behaves  relative  to  the 
conventional  test  plan  for  small  test  numbers. 
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HYPOTHETICAL  SYSTEM  TO  BE  TESTED 


Since  the  cost-effective  test  plan  may  be  used  on  a  variety  of 
different  systems,  it  appeared  reasonable  to  evaluate  the  plan  on 
the  simplest  system.  If  the  benefits  can  be  seen  on  the  simple 
system,  then  they  may  be  even  greater  for  more  complex  systems. 

On  the  other  hand,  if  no  benefits  can  be  seen  on  the  simple  system, 
the  usefulness  of  a  cost-effective  test  plan  may  be  questionable. 

The  configuration  of  the  hypothetical  system  used  to  compare  the 
cost-effective  test  plan  against  the  conventional  test  plan  is  shown  below. 


Exhibit  1 

HYPOTHETICAL  SYSTEM 


The  cost  per  test  of  each  component  and,  therefore,  of  the  entire  system 
is  as  follows: 


Component  1 

$2500 

Component  2 

$3500 

System 

$9500 

Vs. 


MAXIMUS 


If,  for  example,  $95,  000  were  available  for  testing,  the  conven¬ 
tional  test  plan  would  test  the  hypothetical  system  ten  times.  On  the 
other  hand,  the  cost  effective  test  plan  would  spend  only  the  amount 
of  test  dollars  needed  to  attain  the  level  of  risk  achieved  by  the  con¬ 
ventional  test  plan,  presuming  that  it  can  be  done  for  less  than  $95,  000. 

The  predetermined  test  results  for  the  system  were  8  successes 
and  2  failures.  The  component  test  results  resulting  from  the  system 
tests  were  assumed  to  be  as  follows. 

Exhibit  2 

CONVENTIONAL  TEST  PLAN 


Successes 


Failures 


Total  Tests 


Component  1  8  2  10 

Component  2  20  0  20 

The  sequential  plan  used  the  same  component  test  results  except 
that  all  additional  tests  required  of  a  component  were  assumed  to  be 
successes.  The  results  of  the  cost  effective  test  were  as  follows: 

Exhibit  3 

COST-EFFECTIVE  TEST  PLAN 


Successes 


Failures 


Total 


Component  1  12  2  14 

Component  2  2  0  2 

Under  both  test  plans  (with  these  predetermined  test  results  and  the 
prior  distributions  listed  in  Exhibit  4)  the  final  values  of  the  system 
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risk  were  nearly  identical.  However,  as  will  be  shown,  the  cost  of  the 
14  component  tests  conducted  under  the  sequential  test  plan  was  less  than 
half  the  cost  of  10  complete  system  tests.  Thus,  substantial  savings 
may  be  realized. 


IV.  THE  TEST  PLAN  RESULTS 

The  conventional  test  plan,  by  construction,  used  the  entire 
$95,  000  for  its  ten  complete  system  tests.  The  prior  distributions 
placed  on  the  system  and  components  during  both  test  plans  were: 


Exhibit  4 

QUANTITIES  OF  INTEREST 
DERIVED  FROM  PRIOR  DISTRIBUTIONS 


Item 

Bayes'  Estimate 
of  R 

System 

Risk 

(Pseudo) 

Successes 

(Pseudo) 
F  ailures 

Entire  System 

0. 6667 

0.  0171 

7.  00 

3.  00 

Component  1 

0.  8165 

0.  0127 

7.  82 

0.  98 

Component  2 

0.  5716 

0.  0318 

2.  82 

1.  87 

The  risk  of  mis  estimation,  based  solely  on  prior  distributions  before 
either  test  plan  was  implemented,  was  1709.  This  value  is  defined  to 
equal  100,000  times  the  variance  of  the  distribution  of  system  reliability. 

The  risk,  according  to  the  conventional  test  plan  results,  was  862. 
Interestingly,  the  system  risk  was  lower  when  the  system  test  results 
were  used  to  update  the  prior  distribution  of  system  reliability  instead 


of  the  more  common  practice  of  using  the  component  test  results  to 


J 


MAXIMUS 


r 


update  the  prior  distributions  of  component  reliabilities.  Consequently, 
we  used  the  system  test  results,  which  are  shown  in  Exhibit  5  for 
comparison  purposes,  since  they  provided  a  lower  risk. 

Exhibit  5 

QUANTITIES  OF  INTEREST 
DERIVED  FROM  THE  POSTERIOR  DISTRIBUTIONS 

Conventional  Test  Plan 

Bayes'  Estimate  System  (Pseudo)  (Pseudo) 
of  R  Risk  Successes  Failures 


System  Reliability  0.  7273  862  15.00  5.  00 

The  goal  of  this  demonstration  is  to  show  that  a  system  risk  of 
862  can  be  obtained  without  expending  the  full  test  budget  when  a  cost- 
effective  plan  is  used  for  testing  individual  components.  It  will,  in 
fact,  be  shown  that  less  than  half  of  the  test  budget  need  be  spent  to 
attain  this  level  of  risk. 

The  next  exhibit,  Exhibit  6,  summarizes  the  output  from  ABRAM 
used  to  identify  the  cost  savings.  Each  line  of  this  exhibit  shows  the 
necessary  information  for  implementing  the  next  step  of  the  cost  effective 
plan.  For  example,  the  first  line  shows  the  starting  position  of  the  test 
program.  Based  solely  upon  prior  distributions,  the  Bayes'  risk  of 
misestimation  is  1709.  The  second  and  third  columns  from  the  right 
show  the  average  costs  per  unit  reduction  in  risk  that  are  expected  if 
a  particular  component  is  tested.  For  example,  if  component  number  one 
is  tested,  it  is  expected  that  the  $2500  cost  will  produce  72  ($2500/34.  80) 
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units  of  risk  reduction;  whereas,  if  component  two  is  tested,  the  $3500  cost 
is  expected  to  produce  102  ($3500/34.40)  units  of  risk  reduction.  There¬ 
fore,  the  cost-effective  test  plan  tests  component  number  two  first. 

As  shown,  the  first  test  result  was  predetermined  to  be  a  success. 
After  this  result  is  given  to  ABRAM,  the  program  produces  the  calcula¬ 
tions  shown  on  the  second  line. 

The  estimate  of  system  reliability  increases  because  of  the  success 
and  the  risk  decreased  to  1499.  However,  the  risk  decreased  by  more 
than  the  102  units  expected.  The  difference  between  the  expected  de¬ 
crease  and  the  actual  decrease  occurred  because  of  the  randomness 
associated  with  the  test  results.  ABRAM  bases  its  calculations  on 
the  expected  value  of  the  risk  reduction;  sample  realizations  are  anti¬ 
cipated  to  deviate  from  these  expected  values. 

After  the  sixth  test,  for  example,  the  risk  actually  increased.  It 
increased  because  the  Bayes'  estimate  indicated,  before  this  test, 
that  the  reliability  of  the  system  was  over  .  7,  but  then  a  failure 
occurred.  This  test  result  conflicts  to  some  degree  with  the  pretest  ex¬ 
pectations  accounting  for  the  increased  risk. 

After  the  13th  test,  a  level  of  risk  nearly  equal  to  that  of  the  con¬ 
ventional  test  plan  was  reached  (890  as  compared  to  862).  However,  the 
cost-effective  plan  required  only  $34,  500  of  the  $95,000  test  budget.  Since 


the  stated  goal  of  this  demonstration  was  to  attain  at  least  as  low  a  risk 
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using  the  cost-effective  plan  as  achieved  by  the  conventional  plan,  one 

more  test  was  conducted.  The  first  component  was  tested  again  in 
the  fourteenth  test,  since  it  was  cost-effective  to  do  so,  and  the  test 
was  predetermined  to  be  a  success.  The  total  cost,  then,  was  $37,  500, 
a  61%  reduction  from  $95,  000. 

V.  CONCLUSIONS 

The  substantial  savings  that  were  achieved  in  the  system  in  Exhibit 
1  resulted,  in  part,  from  components  connected  in  a  parallel.  A  complete 
system  test  is  wasteful,  in  this  case,  because  less  information  is  needed 
to  be  precise  about  the  second  serial  subsystem,  which  consists  of  the 
two  parallel  components.  Hence,  when  there  are  subsystems  with  com¬ 
ponents  in  parallel,  the  cost-effective  test  plan  should  produce  reliability 
assessment  information  at  less  cost. 

Another  advantage  of  the  cost-effective  test  plan  is  that  it  takes 
into  account  the  cost  differentials  involved  in  testing  different  compon¬ 
ents.  Two  (approximately)  equally  reliable  but  different  components 
may  provide  nearly  the  same  test  information  about  the  system  relia¬ 
bility.  However,  it  makes  sense  to  test  one  of  the  components  more 
than  the  other  if  test  budgets  are  constrained.  The  cost-effective  test 
plan  does  this,  but  the  conventional  test  plan  does  not. 

Several  qualifications  should  be  made  concerning  the  preceding 
statements.  First,  the  example  is  only  that- -an  example.  The 
savings  of  61%  can  not  be  anticipated  in  every  situation.  Second, 
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individual  component  tests  do  not  provide  information  about  the  work¬ 
manship  connecting  the  different  components.  If  there  is  realistic  con¬ 
cern  about  the  dependency  among  component  reliabilities,  then  some 
complete  systems  or  subsystems  tests  should  be  conducted.  Third,  a  more 
practical  test  plan  would  test  more  than  one  individual  component  at  a 
time.  It  is  not  realistic,  for  example,  to  expect  tests  to  be  conducted  one 
at  a  time,  awaiting  the  results  of  a  previous  test.  Fourth,  the  methodology 
currently  does  not  consider  fixed  set-up  costs  for  testing.  If  a  test 
facility  has  to  be  rented,  for  example,  a  particular  component  may  have 
a  high  set-up  cost.  Finally,  it  should  be  noted  that  since  test  results 
are  random  and  the  cost  effective  test  plan  uses  these  results  in  a 
sequential  manner,  there  is  always  a  chance  that  the  cost-effective  plan 
will  actually  reduce  the  risk  less  than  the  conventional  plan.  However, 
this  should  be  a  rare  occurrence  if  the  number  of  tests  conducted  is 
large. 

In  sum,  this  short  exercise  has  raised  almost  as  many  questions 
as  it  has  answered.  A  true  test  of  the  methodology  will  require  taking 
an  actual  system  using  actual  test  and  cost  data,  and  determining  the 
savings  that  are  possible.  We  would  not  be  surprised,  however,  to 
find  that  these  savings  are  substantial. 
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PART  3 

USER'S  MANUAL 


AUTOMATED  BAYESIAN  RELIABILITY  ASSESSMENT  MODEL 

(ABRAM) 


ABRAM  is  a  general  purpose  FORTRAN  IV  computer  software  program 
which  provides  a  technical  or  nontechnical  user  the  capability  to  employ 
Bayesian  reliability  assessment  techniques  in  assessing  the  reliability  of 
multi- component  systems.  ABRAM  has  several  features  which  distinguish 
it  from  traditional /classical  reliability  assessment  programs: 

•  The  user  need  not  know  how  to  construct  a  mathematical 
model  which  relates  the  reliabilities  of  the  components  of 
a  multi- component  system  to  the  reliability  of  the  system. 

ABRAM  generates  the  mathematical  model  internally. 

•  ABRAM  allows  the  user  to  specify  prior  distributions  of 
reliability  at  the  system  level,  the  subsystem  level,  the 
component  level,  or  at  all  three  levels.  Consistency  in 
selection  of  prior  distributions  is  assured  by  the  program. 

•  ABRAM  permits  the  combination  of  system  test  results  and 
component  test  results  in  the  same  assessment.  This  feature 
allows  the  user  to  apply  all  the  data  on  a  particular  system 

to  determine  estimates  of  reliability. 

•  An  optimal  sequential  testing  methodology  is  programmed 
into  ABRAM.  This  feature  allows  the  user  to  obtain  test 
data  at  the  most  cost-effective  rate  to  acquire  knowledge  of 
the  reliability  of  the  system. 

Thus,  ABRAM  is  a  state-of-the-art  computer  program  which  provides 
a  number  of  options  to  the  user  in  employing  Bayesian  reliability  assess¬ 
ment  techniques.  The  purpose  of  this  manual  is  to  provide  instructions 


necessary  to  use  the  program. 
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The  manual  is  organized  into  six  sections: 

•  Review  of  the  Bayesian  Approach  to  Reliability  Assessment 

•  Instructions  for  Coding  the  Component  Configuration  in  the 
System 

•  Guidelines  for  Specifying  Prior  Distributions  of  Reliability 

•  Instructions  for  Formatting  Input  Data 

•  Interpretation  of  the  Output  of  ABRAM 

•  Listing  of  Program  Code  of  ABRAM 

I.  REVIEW  OF  BAYESIAN  APPROACH  TO  RELIABILITY  ASSESSMENT 


The  Bayesian  approach  to  statistical  estimation  is  based  on  the  notion 
that  probability  can  be  interpreted  as  the  degree  of  belief  in  an  event,  as 
opposed  to  the  frequency  with  which  an  event  occurs.  In  the  case  of  relia¬ 
bility  assessment,  the  event  is  defined  as  the  successful  operation  of  the 
system  or  component.  Although  the  component  or  system  may  have  a  fixed 
reliability  (or  probability  of  success),  the  value  of  the  particular  reliability 
is  unknown  to  the  analyst.  The  analyst,  however,  may  have  incomplete 
knowledge  about  the  reliability  of  a  particular  component  or  of  the  system 
which  he  desires  to  express  in  a  formal  way. 

The  analyst  using  the  Bayesian  approach  specifies  his  degree  of  belief 
that  the  reliability  of  the  system  or  component  takes  on  specific  values 
between  zero  and  one.  The  mechanism  that  is  used  to  describe  this  degree 
of  belief  is  a  probability  density  function  which  defines  the  probability 


(subjective)  that  the  true  reliability  of  the  component  or  system  falls  in  a 


particular  interval  of  reliability  values.  Thus,  the  Bayesian  approach 
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incorporates  in  a  formal  way  the  analyst's  pretest  opinions  about  the  com¬ 
ponent  and  system  reliabilities. 

The  following  exhibit  illustrates  three  different  probability  density 
functions  of  reliability.  The  uniform  distribution,  labeled  I,  may  be  con¬ 
sidered  neutral,  in  that  every  reliability  is  perceived  as  being  equally  likely 
before  test  results  are  observed.  That  is,  the  analyst  does  not  believe  that 
he  knows  enough  to  favor  any  particular  interval  of  reliability  over  any  other 
interval.  The  second  distribution,  labeled  II,  suggests  more  information, 
because  the  analyst  is  expressing  that  the  true  value  of  reliability  is  believed 
to  be  between  .  50  and  .  90.  The  distribution  labeled  III  provides  the  most 
information,  because  it  reflects  the  analyst's  belief  that  the  true  reliability 
is  between  .  70  and  .  90. 


Exhibit  1 


PROBABILITY  DISTRIBUTION  FUNCTIONS 
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The  form  of  the  prior  distribution  of  reliability  can  be  specified  para¬ 
metrically  when  the  beta  family  of  distribution  is  used  with  two  parameters 
termed  pseudo -successes  and  pseudo -failures.  The  prefix  pseudo  is  used 
because  the  values  are  not  really  successes  and  failures;  however,  the 
parameters  of  the  distribution  change  exactly  as  if  the  same  number  of 
successes  and  failures  were  observed  as  test  results.  Hence,  the  values 
of  the  parameters  are  commonly  interpreted  as  pseudo- successes  and 
ps  eudo-  failur  e  s . 

The  following  exhibit  shows  the  parameters  of  the  three  density 
functions. 

Exhibit  2 

PARAMETERS  OF  DENSITY  FUNCTIONS 

A  B 

Density  Function  Pseudo-Successes  Pseudo -Failures 

I  0  0 

II  4  1 

HI  8  2 

The  functional  form  of  the  density  function  is  as  follows: 
f(R)  =  K*  RA  *  (1-R)B  , 

where  K  is  a  constant,  A  the  number  of  pseudo-successes,  and  B,  the 
number  of  pseudo-failures 
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The  attractive  feature  of  the  beta  family  of  distributions,  when  used 
as  prior  distributions  for  pass-fail  testing,  is  that  when  test  results  are 
obtained,  they  can  simply  be  added  to  the  number  of  pseudo- successes  and 
pseudo-failures  to  obtain  the  revised  distribution  of  degree  of  belief.  For 
example,  the  distribution  III  would  also  be  the  posterior  distribution  of 
reliability  had  the  analyst  started  with  the  initial  (or  prior)  distribution  I 
and  then  observed  8  successes  and  2  failures  as  test  results. 

In  assessing  the  reliability  of  a  multi-component  system,  the  analyst's 
degree  of  belief  about  the  reliability  of  each  component  and  the  system 
must  be  specified  either  by  the  analyst  or  by  ABRAM.  That  is,  a  prior 
distribution  of  reliability  must  be  specified  for  the  system  and/or  each 
component.  ABRAM  assigns  a  uniform  distribution  as  the  prior  dis¬ 
tribution  of  the  system,  unless  the  analyst  specifies  otherwise.  Prior 
distributions  for  component  reliabilities  that  are  not  specified  are  as¬ 
signed  by  ABRAM  subroutines,  which  allocate  uncertainty,  as  measured 
by  the  variance  of  the  distribution,  evenly  over  all  components. 

It  should  be  noted  that,  in  general,  the  parameters  pseudo-successes 
and  pseudo-failures  need  not  be  integer  valued,  as  must  the  test  results 
describing  successes  and  failures.  Moreover,  these  parameter  values  can 
be  negative  as  long  as  they  are  greater  than  minus  one;  for  example,  values 
of  A  and  B  of  -.  99  are  permissable.  The  lower  the  values  of  A  and  B, 
the  easier  it  will  be  for  test  results  to  mask  or  overpower  the  original 
selection  of  A  and  B.  Thus,  if  A  and  B  are  initially  0.  5  and  0t  0,  say,  J 
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the  posterior  distribution  that  would  result  after  10  successful  tests 
would  be  10.  5  and  1.  0.  Thus,  the  test  results  are  much  more  important 
than  the  prior  specification  of  the  distribution.  If  high  values  of  A  and  B 
are  chosen  to  characterize  the  prior  distribution  of  reliability,  it  will 
be  more  difficult  for  the  test  results  to  "wash  out"  the  implications  of 
the  prior  specification. 

The  Bayesian  approach  also  consists  of  assigning  a  loss  that  will 
result  if  the  true  reliability  is  misestimated.  The  most  commonly  assumed 
form  of  the  loss  function  is  the  squared  error  loss  function.  Thus,  if  R  is 
the  true  reliability  and  ^  is  the  estimate,  the  loss  is  proportional  to  (R-R/7 
The  risk  is  defined  as  the  expected  loss  and  is  calculated  using  the  proba¬ 
bility  distribution  function  of  reliability. 

1 

RISK  =  /  f(R)  C&-R)2  dR. 

0 

The  Bayes'  estimate  of  reliability  is  the  one  that  minimizes  the  risk. 


For  a  squared  error  loss  function,  the  Bayes'  estimate  is  the -mean  of  the 
distribution  and  the  risk  is  directly  proportional  to  the  variance  of  the 
distribution.  The  first  two  moments  of  any  distribution,  then,  provide  the 
necessary  information  for  a  complete  Bayesian  assessment.  From  this 
information  the  risk  can  be  computed.  ABRAM  takes  advantage  of  these 
technical  facts  and  works  only  with  the  first  two  moments  of  each  distri¬ 
bution  in  developing  estimates  of  system  reliability. 


MAXIMUS 


"\ 


H.  INSTRUCTIONS  FOR  CODING  COMPONENTS 


Because  of  the  complexity  of  the  calculations  involved  in  computing 
prior  and  posterior  distributions  of  component  and  system  reliabilities, 
we  have  developed  an  automatic  math  model  generator  as  one  of  the 
subroutines  of  ABRAM,  This  subroutine  will  construct  a  model  of  the 
system  if  the  input  data  are  coded  according  to  the  scheme  described  in 
this  section.  Thus,  the  analyst  does  not  have  to  input  a  complex  equation 
relating  component  reliabilities  to  the  system  reliability.  Before 
graphically  illustrating  the  code,  we  give  a  brief  description. 

The  position  of  each  component  in  the  system  is  represented  by  a 
five-digit  code.  The  first  digit  from  the  left  identifies  the  serial  sub¬ 
system  to  which  the  component  belongs.  That  is,  all  components  belonging 
to  the  first  subsystem,  for  example,  have  codes  in  the  10000  series. 

The  second  digit  identifies  the  parallel  path  of  the  serial  subsystem 
to  which  the  component  belongs.  The  third  digit  identifies  the  serial 
group  of  the  parallel  path.  The  fourth  identifies  the  parallel  subpath 
of  the  serial  group,  and  the  fifth  the  serial  subgroup  of  the  parallel  sub¬ 
path.  If  the  digit  is  zero,  then  no  such  serial  or  parallel  subgroup 
component  exists. 

For  example,  consider  the  following  system: 

) 


Exhibit  3 


SERIAL  SUBSYSTEMS 


System  gets 
a  code  of 
000000 - ^ 


The  code  10000  refers  to  the  first  subsystem;  the  code  20000  refers  to  the 
second  subsystem.  The  two  parallel  paths  in  the  second  subsystem  are 
denoted  21000  and  22000. 

Assume  that  the  first  parallel  path  of  the  second  subsystem  is  composed  of 
additional  groups  of  components  in  series  as  shown  below. 


Exhibit  4 

PARALLEL  PATH 


These  serial  groups  would  be  coded  21100  and  21200  to  designate  their 
position  in  the  system.  Suppose  further  that  the  second  serial  group  in 


First  Serial  Subsystem  Second  Serial  Subsystem 


t  i 


10000  20000 

!r —  21000—  | 

,1 —  22000)—  ' 

^21000,  which  we  call  21200,  had  three  parallel  subpaths  as  follows:  J 
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In  summary,  the  hypothetical  system  described  would  appear  as 
shown  below.  Note  that  each  component  has  a  code  which  identifies: 

•  the  serial  subsystem; 

•  the  parallel  path  of  the  subsystem; 

•  the  serial  group  of  the  parallel  path; 

•  the  parallel  subpath  of  the  serial  group; 

•  the  serial  subgroup  of  the  parallel  subpath. 

Exhibit  7 
TOTAL  SYSTEM 


,  i - 1  21000  • 


21210 


21200 


21100 


21220 


21222 


21230 


10000 


2000 

-  22000 
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Thus,  any  system  with  five  levels  of  complexity  can  be  described  by  the 
five-digit  code. 

In  coding  these  components,  all  higher  level  codes  should  also  be 
present.  For  example,  although  a  single  component  comprises  the  first 
serial  subsystem  (code  10000),  several  components  comprise  the  second 
serial  subsystem.  Although  the  code  20000  does  not  appear  explicitly,  the 
analyst  should  specify  it  for  completeness.  However,  ABRAM  will  provide 
the  higher  level  codes  if  they  are  not  specified. 

The  following  exhibit  shows  the  input  codes  for  the  system  just 
described. 


Exhibit  8 


HYPOTHETICAL  SYSTEM  CODES  fc  COMPONENTS 


1 

Code 

Component  Number 

1 

00000 

* 

2 

10000 

1 

3 

20000 

* 

4 

21000 

* 

5 

21 100 

2 

6 

21200 

* 

7 

21210 

3 

8 

21220 

# 

9 

21221 

4 

10 

21222 

5 

11 

21230 

3+ 

12 

22000 

6 

*  Signifies  that  the  code  does  not  refer  to  an  actual  component,  but  to 
an  aggregation  of  components. 


+  This  component  is  assumed  to  be  identical  to  the  component  coded 
by  21210  and,  therefore,  its  reliability  will  be  the  same. 
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The  code  00000  refers  to  the  entire  system  and  must  always  be 

present.  Because  only  one  digit  is  currently  allowed  to  reference  each 

level,  up  to  9  serial  subsystems  can  be  accommodated. 

III.  GUIDELINES  FOR  SPECIFYING  A  PRIOR  DISTRIBUTION  OF 
RELIABILITY 

This  section  of  the  manual  is  organized  into  three  parts: 

•  specifying  prior  distributions  at  the  component, 
subsystem,  or  system  level; 

•  some  ideas  for  specifying  a  prior  distribution 
when  dealing  with  an  independent  contractor; 

•  some  ideas  for  helping  to  specify  a  prior  distri¬ 
bution  for  internal  use. 

A.  SPECIFYING  PRIOR  DISTRIBUTIONS  AT  THE  COMPONENT. 

SUBSYSTEM.  OR  SYSTEM  LEVEL 

Taking  full  advantage  of  prior  knowledge  about  the  component,  sub¬ 
system,  and  system  reliabilities  enables  the  analyst  to  make  decisions 
with  the  least  risk.  Therefore,  it  should  be  helpful  to  specify  a  prior 
distribution  for  each  component,  subsystem,  and  system  where  there  is 
prior  knowledge.  The  level  in  the  system  where  these  prior  distributions 
are  specified  should  take  into  consideration  the  importance  of  these  levels 
in  the  analysis.  For  example,  if  a  decision  is  to  be  made  on  accepting  a 
system  from  a  contractor,  it  may  be  useful  to  consider  placing  a  uniform 
prior  distribution  on  the  system  reliability.  If,  on  the  other  hand,  a  deci¬ 
sion  is  to  be  made  about  accepting  a  few  components  from  a  contractor,  it 
may  be  useful  to  consider  placing  a  uniform  prior  distribution  on  these 
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components.  When  the  responsibility  for  accepting  various  levels  of  the 
system  is  delegated  to  different  individuals  or  organizations,  a  group  con¬ 
sensus  might  be  required  on  what  prior  to  select.  We  have  no  firm  recom¬ 
mendations  for  resolving  different  organizational  viewpoints  except  that 
agreement  on  the  prior  distribution  of  system  reliability  is  obviously  the 
most  important. 

A  special  feature  of  ABRAM  is  that  it  will  generate  prior  distributions 
for  the  components  or  subsystems  of  the  system  that  the  analyst  does  not 
specify.  Since  mathematical  rules  governing  the  system  operation  do  not 
allow  completely  free  choice  of  prior  distributions,  ABRAM  will  generate 
the  necessary  priors  consistent  with  every  mathematical  rule.  This  is  a 
useful  feature  if  the  analyst  chooses  not  to  specify  a  complete  set  of  prior 
distributions.  For  example,  consider  the  following  system: 

Exhibit  9 

HYPOTHETICAL  SYSTEM 
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Suppose  prior  distributions  are  specified  for  the  system  (00000),  the  first 
serial  subsystem  (10000),  and  the  first  component  of  the  second  serial 
subsystem  (21000).  Then  ABRAM  automatically  assigns  the  appropriate 
prior  for  the  second  serial  subsystem  (20000)  and  its  second  component 


(22000).  The  output  listing  of  ABRAM  provides  the  parameters  of  the  prior 
distributions  internally  assigned  so  the  analyst  can  check  them  for  reason¬ 
ableness. 

B.  SOME  IDEAS  FOR  SPECIFYING  A  PRIOR  DISTRIBUTION  WHEN 


DEALING  WITH  AN  INDEPENDENT  CONTRACTOR 


The  best  procedure  is  to  assign  prior  distributions  which  reflect 
actual  knowledge  about  the  reliability  of  the  system  and  its  components. 

In  this  way  less  risk  will  result  after  testing  the  system.  However,  if  an 
outside  contractor  is  involved,  a  consensus  should  be  reached  about  the 
prior  distribution  of  the  reliability  of  the  system.  It  may  be  in  the  govern¬ 
ment's  best  interest  not  to  let  the  contractor  choose  the  prior  distribution 
at  the  system  level.  After  all,  the  contractor  has  a  vested  interest  in 
demonstrating  the  high  reliability  of  his  system.  In  such  cases,  it  would 
be  safe  to  place  a  uniform  prior  on  the  system  (or  the  highest  level  of  the 
system  with  which  the  contractor  has  a  direct  connection).  This  allows  the 
test  data  to  have  an  effect  in  the  analysis.  When  several  different  con¬ 
tractors  are  making  components  for  the  system,  and  different  DoD  offices 
have  responsibility  for  deciding  whether  or  not  to  accept  them,  each  office 


should  makes  its  own  decisions  about  prior  specification. 
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When  only  one  analyst  is  deciding  whether  or  not  to  accept  the  con¬ 
tractor's  components,  the  following  rule  of  thumb  may  be  useful.  The  dat- 
will  have  significant  impact  on  the  systems  analysis  if  the  uniform  prior  is 
placed  on  the  system,  a  moderate  impact  if  the  uniform  prior  is  placed  on 
each  subsystem,  and  the  least  impact  if  the  uniform  prior  is  placed  on  all 
the  components.  In  the  latter  two  cases,  incidentally,  the  system  prior 
will  be  skewed  toward  zero. 


C.  SOME  GUIDELINES  FOR  SPECIFYING  PRIOR  DISTRIBUTIONS  FOR 
INTERNAL  USE 

The  key  point  for  the  analyst  to  consider  in  choosing  a  prior  distribu¬ 
tion  is  the  sensitivity  of  the  posterior  assessment  to  the  test  results. 

If  there  are  recent  data  available  about  the  reliability  of  the  system 
and  components,  they  can  be  used  to  specify  the  prior  distribution.  ABRAM 
will  accept  the  given  number  of  successes.  A,  and  failures,  B,  of  the 
system,  subsystem,  and  components  as  prior  specifications  and  compute 
a  consistent  set  of  prior  distributions  for  the  remaining  components  and 
subsystems. 

When  there  is  not  recent  data  available,  the  analyst  may  specify 
his  prior  opinions  in  terms  of  pseudo-successes.  A,  and  pseudo-failures, 
B,  which  reflect  his  degree  of  belief  about  the  reliability.  The  following 
rules  of  thumb  may  be  helpful  in  specifying  a  prior  distribution: 

•  When  the  analyst  is  reasonably  convinced  about  what  the 
component,  subsystem,  or  system  reliabilities  are,  he 
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might  specify  his  prior  by  choosing  A  4  B  =  20  and  (A/ 20)  equal 
to  his  best  point  estimate  of  the  component  and  system  reliabilities. 
This  specification  reduces  the  impact  of  any  future  test  results  on 
the  analysis.  For  example,  if  he  is  reasonably  certain  the  relia¬ 
bility  is  0.  8,  he  may  set  A  =  16  and  B  =  4.  (Again,  it  may  be 
useful  to  recall  that  the  system  prior  distribution  is  the  most 
important  to  specify.  ) 

•  When  the  analyst  has  some  idea  about  what  the  component,  sub¬ 
system,  or  system  reliabilities  are,  he  might  specify  his  prior 
distribution  by  setting  A  4  B  =  5  and  (A/5)  equal  to  his  best  point 
estimate  of  the  reliability.  This  choice  of  prior  distribution  will 
give  the  data  a  moderate  impact  on  the  analysis.  For  example,  if 
he  is  moderately  certain  that  the  reliability  is  .  8,  he  might  set 

A  =  4  and  B  =  1. 

•  When  the  analyst  is  not  at  all  certain  about  what  the  system  and 
component  reliabilities  are,  he  might  specify  a  uniform  prior 
distribution  for  the  system  by  setting  A  =  0,  B  =  0.  This  speci¬ 
fication  will  give  future  test  results  a  great  impact  in  the  analysis. 

Regardless  of  how  the  prior  distributions  are  specified,  certain 
assignment  rules  must  be  followed  or  ABRAM  will  print  the  error  message: 
"Component  and  System  Priors  are  Inconsistent.  "  These  rules  are  that 
all  serial  subsystems  must  be  less  reliable  than  any  of  their  serial  com¬ 
ponents  and  all  parallel  subsystems  must  be  more  reliable  than  any  of 
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their  components.  Moreover,  the  variance  of  any  distribution  must  be 
greater  than  zero. 

For  example,  consider  the  following  two  subsystems. 

Exhibit  1 0 

SUBSYSTEM  EXAMPLES 


SUBSYSTEM  A 


SUBSYSTEM  B 


If  the  first  (or  second)  moment  of  the  prior  distribution  of  the  reliability 
of  subsystem  A  is  greater  than  the  first  (or  second)  moment  of  the  prior 
distributions  of  components  one  or  two,  ABRAM  will  print  the  error 
message.  If  the  moments  of  the  prior  distribution  of  the  reliability  of 
subsystem  B  are  less  than  either  of  the  moments  of  the  prior  distribu¬ 
tions  of  the  reliabilities  of  components  one  or  two,  ABRAM  will  print  the 
error  message.  If  the  variance  of  any  component  or  system  distribution 


is  negative,  ABRAM  will  print  the  error  message. 


J 
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IV  INSTRUCTIONS  FOR  FORMATTING  INPUT  DATA 

The  procedure  for  formatting  input  data  for  ABRAM  will  be  explained 


using  the  following  system: 


Exhibit  1 1 


SYSTEM  EXAMPLE 


00000 


10000 


_  20000 

-  21000  - 

2 


22000 


COST  OF  TESTING:  Component  1  =  $250;  Component  2  =  $350 
This  section  presents  the  steps  that  should  be  used  to  utilize  ABRAM 
effectively.  Three  exhibits  are  presented:  Exhibit  12  shows  the  proper 
input  format  for  ABRAM,  and  Exhibits  13  and  14  show  the  proper  input 
format  for  the  hypothetical  system  in  Exhibit  1 1  above. 

STEP  1:  SPECIFY  THE  MODE  IN  WHICH  ABRAM  IS  TO  WORK 


Since  analyzing  test  data  to  compute  estimates  of  reliability  does 
not  require  calculating  a  cost-effective  test  plan,  ABRAM  can  operate 
in  two  modes.  The  choice  of  modes  allows  ABRAM  to  be  used  most 
efficiently. 
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Exhibit  1 3 


CORRECT  INPUT  DATA  FORMAT  -  SIMULTANEOUS  MODE 


Card 

Code 

1. 

1 

2. 

005002 

3. 

0000000000 

4. 

1000001250 

5. 

2000000000 

6. 

2100002350 

7. 

2200002350 

8. 

02 

9. 

00000007002 

10. 

10000005000 

II. 

02 

12. 

01007000 

13. 

02008001 

Exhibit  14 

CORRECT  INPUT  DATA  FORMAT  -  SEQUENTIAL  MODE 

Card 

Code 

1. 

0 

2. 

005002 

3. 

0000000000 

4. 

1 000001250 

5. 

2000000000 

6. 

2100002350 

7. 

2200002350 

8. 

02 

9. 

00000007002 

10. 

1 0000005000 

11. 

01007000 

12. 

02008001 

13. 

000000 

J 
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ABRAM  can  aid  in  finding  both  the  best  sequential  test  plan  and  the 
Bayes'  estimates  of  system,  subsystem,  and  component  reliabilities. 

•  The  simultaneous  mode  provides  the  usual  and  necessary  statis¬ 
tical  analysis  for  estimating  the  reliabilities  without  calculating 
a  cost-effective  test  plan.  This  is  the  traditional  analysis  that 
is  performed  after  all  testing  is  completed  and  the  results  are 
available.  The  simultaneous  mode  is  specified  by  MODE  =  1. 

•  The  sequential  mode  provides  a  cost-effective  test  plan  developed 

sequentially,  as  well  as  the  same  statistical  capabilities  as  the 

simultaneous  mode.  The  sequential  mode  is  specified  by  MODE  =  0. 

STEP  2:  SPECIFY  BOTH  THE  NUMBER  OF  CONFIGURATION  CODES  AND 
THE  NUMBER  OF  DISTINCT  COMPONENTS 

After  ABRAM  is  given  the  mode  specification,  it  will  generate  a  math 
model  for  the  system.  In  order  to  do  this  effectively,  it  must  be  supplied 
with  the  proper  configuration  codes  and  component  numbers  as  described 
in  Section  IL  ABRAM  reads  exactly  the  number  of  configuration  codes 
specified  by  MAX.  For  the  system  in  Exhibit  11,  MAX  =  5. 

In  addition,  ABRAM  sets  up  a  model  having  the  number  of  distinct 
components  =  ICMAX.  Since  there  are  only  two  distinct  components  of 
the  system  in  Exhibit  11  (the  one  component  comprising  the  first  serial 

subsystem  and  the  two  identical  components  comprising  the  second  serial 
subsystem)  ICMAX  =  2. 
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STEP  3:  SPECIFY  THE  COMPONENT  CONFIGURATION  CODES, 
COMPONENT  NUMBER.  AND  COST  OF  TESTING  EACH 
COMPONENT 

ABRAM  generates  the  proper  math  model  having  at  least  the  number 
of  subsystems  specified  by  Step  2.  The  component  configuration  codes 
indicate  where  each  component  is  located,  and  the  component  identification 
numbers  indicate  which  components  are  distinct.  Thus,  the  identical 
components  will  have  different  configuration  codes  but  the  sample  compon¬ 
ent  identification  number.  (Note  that  identical  components  should  have 
identical  prior  distributions  also.)  In  the  sequential  mode.  Mode  0, 

ABRAM  finds  the  cost-effective  test  plan  using  the  inputted  amount  that 
it  costs  to  test  each  component. 

For  the  system  shown  in  Exhibit  11,  the  codes  will  be: 


Exhibit  1 5 

COMPLETE  SET  OF  CODES 


IC 

Configuration  Codes 
00000 
10000 
20000 
21000 
22000 


ICN 

Component  Number 
0 
1 
0 
2 
2 


I  CO  ST 

Cost  of  Testing 
0 

250 

0 

350 

350 


(The  syst^-n  and  subsystems  which  have  more  than  one  component  are 


given  the  component  number  0.  ) 
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ABRAM  will  generate  the  correct  math  model  ever,  if  it  is  only  given 


the  following  codes: 


Configuration  Codes 


10000 


Exhibit  16 

REDUCED  SET  OF  CODES 
ICN 

Component  Number 


ICCST 

Cost  of  Testin 


21000 


22000 


However,  ABRAM  allows  user- specified  prior  distributions  only  for 
user-specified  configuration  codes.  Thus,  the  greatest  benefit  from 
ABRAM  can  be  obtained  by  specifying  all  the  configuration  codes. 

STEP  4:  SPECIFY  THE  NUMBER  OF  PRIOR  DISTRIBUTIONS 


The  prior  distributions  can  be  specified  for  the  system,  subsystem, 
and  component  reliabilities  using  the  configuration  codes.  NUMPRI  = 

number  of  prior  distributions  to  be  specified.  For  each  prior  distribu¬ 
tion,  the  user  must  also  specify  the  configuration  code  and  the  pseudo¬ 
successes  and  the  pseudo-failures  as  shown  in  cards  8,  9,  and  10  of 
Exhibit  13. 

Full  advantage  of  knowledge  of  the  system's,  subsystem's,  and 
components'  recent  reliability  performance  can  be  obtained  by  specifying 
prior  distributions  according  to  the  guidance  provided  in  Section  III  of 
this  Manual. 
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SIMULTANEOUS  MODE  ONLY):  SPECIFY  THE  NUMBER  OF 


*' .  *v<V^Y 
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V.  INTERPRETATION  OF  OUTPUT  LISTING  OF  ABRAM 


In  order  to  under  stand  the  output  from  ABRAM,  it  should  be  helpful 
to  study  the  two  printouts  at  the  end  of  this  section.  They  are  ABRAM's 
response  to  the  simultaneous  mode  input  of  Exhibit  13  and  the  sequential 
mode  input  of  Exhibit  14. 

The  first  table  of  output  from  ABRAM  is  the  EDITED  INPUT  which 
consists  of  the  analyst's  input  information  and  some  information  generated 
internally  by  ABRAM. 

The  REFERENCE  INDEX  is  an  internally  generated  code  that  gives 
each  component,  subsystem,  and  system  a  numerical  name.  ABRAM  uses 
this  name  to  communicate  to  the  analyst  which  item  in  the  system  is  being 
addressed.  These  names  are  listed  in  a  logical  order;  consecutive  system 
components  have  consecutive  names. 

The  CONFIGURATION  CODE  is  the  five-digit  code  described  in 
Section  III  that  may  have  been  provided  as  input.  If  a  subsystem  code  was 
not  provided  as  input,  ABRAM  generates  it  in  its  logical  position. 

The  COMPONENT  NUMBER  LIST  gives  the  listing  of  identical  com¬ 
ponents  that  was  provided  by  the  analyst  as  input.  ABRAM  assigns  the  com¬ 
ponent  number  0  to  the  system  and  all  subsystems  not  composed  of  a  single 
component. 

The  COST  OF  TESTING  gives  the  cost  of  testing  each  component 
provided  by  the  analyst  as  input.  Since  ABRAM  does  not  suggest  testing 
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calculations  on  the  expected  system  risk  incurred  by  testing  each  component 
(EXPECTED  SYSTEM  RISK  IF  TESTED).  The  cost-effective  test  plan 
identifies  the  component  that  produces  the  greatest  reduction  in  the  system 
risk  per  dollar  spent  (or  COST  OF  TESTING/CHANGE  FROM  CURRENT 
RISK  =  COST  PER  UNIT  CHANGE  IN  RISK). 

The  next  table  that  ABRAM  gives  are  the  component  test  results.  (In 
the  sequential  mode.  Mode  0,  they  are  given  one  at  a  time.  )  They  were 
provided  to  ABRAM  as  input. 

The  last  table,  ESTIMATES  OF  RELIABILITY  DERIVED  FROM 
POSTERIOR  DISTRIBUTIONS,  provides  the  standard  Bayesian  statistical 
reliability  analysis.  The  prior  distributions  (both  provided  as  input  or 
internally  generated)  are  combined  with  the  test  results  to  provide  this 
information.  Each  entry  in  this  table  has  the  same  interpretation  as  the 
entries  in  the  table  for  prior  distributions  except  that  these  entries  are 
based  on  test  result  data  and  prior  distributions. 
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Exhibit  19 

OUTPUT  FROM  MODE  1 


AUTOMATED  BAYESIAN  RELIABILITY  ASSESSMENT  MODEL 


EDITED  INPUT  DATA 


REFERENCE 

CONFIGURATION 

COMPONENT 

COST  (»>  OF 

INDEX 

CODE 

NUMBER 

TESTING 

I 

0 

0 

0 

2 

lOOOO 

1 

230 

3 

20000 

0 

0 

4 

21000 

2 

330 

S 

22000 

2 

330 

SPECIFICATION  OF  PRIOR  DISTRIBUTIONS  AND/OR 
SYSTEM/SUBSYSTEM  TEST  RESULTS 


REFERENCE 

BEST  ESTIMATE 

VARIANCE 

(PSEUDO) 

(PSEUDO) 

INDEX 

OF  R 

OF  R 

SUCCESSES 

FAILURES 

1 

0.7273 

0.01A3 

7.00 

2.00 

2 

0.8371 

0.0133 

3.00 

0.0 

ESTIMATES 

OF  RELIABILITY 

DERIVED  FROM 

PRIOR  DISTRIBUTIONS 

REFERENCE 

BEST  ESTIMATE 

VARIANCE 

(PSEUDO) 

(PSEUDO) 

INDEX 

OF  R 

OF  R 

SUCCESSES 

FAILURES 

1 

0.7273 

0.0143 

7.00 

2.00 

2 

0.8571 

0.0153 

3.00 

0.00 

3 

0.8483 

0.0073 

13.00 

1.30 

4 

0.4108 

0.0224 

4.82 

2.71 

3 

0.4108 

0.0224 

4.82 

2.71 

s'. 
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Exhibit  19 

OUTPUT  FROM  MODE  1 


(Continued) 


COMPONENT  TEST  RESULTS 


COMPONENT  SUCCESSES  FAILURES 


ESTIMATES  OF  RELIABILITY  DERIVED  FROM  POSTERIOR  DISTRIBUTIONS 


REFERENCE  BEST  ESTIMATE  VARIANCE 
INDEX  OF  R  OF  R 


(PSEUDO)  (PSEUDO) 


OF  R 

OF  R 

SUCCESSES 

FAILURES 

0 . 8686 

0.0050 

17.62 

1.83 

0*9264 

0.0044 

12.00 

0.00 

0.9334 

0.0013 

40.00 

1.63 

0.7438 

0.0097 

12.62 

3.71 

0.7436 

0.0097 

12. B2 
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Exhibit  20 

OUTPUT  FROM  MODE  0 


AUTOMATED  DATCSIAN  RELIABILITY  ASSESSMENT  MODEL 

EDITED 

input  data 

REFERENCE  CONFIGURATION 

COMPONENT 

COST  (»> 

OF 

INDEX 

CODE 

NUMBER 

TESTING 

1 

0 

0 

0 

2 

10000 

1 

250 

3 

20000 

0 

0 

4 

21000 

2 

350 

s 

22000 

2 

350 

SPECIFICATION  OF 

PRIOR  DISTRIBUTIONS  AND/OR 

SrSTEH/SUOSTSTEM  TEST  RESULTS 

REFERENCE 

BEST  ESTIHATC 

VARIANCE 

( PSEUDO  > 

(PSEUDO) 

INDEX 

OF  R 

OF  R 

SUCCESSES 

FAILURES 

1 

0.7273 

0.0145 

7.00 

2.00 

2 

0.SS71 

0.0153 

5.00 

0.0 

ESTIMATES 

OF  RELIABILITY 

DERIVED  FROM 

PRIOR  DISTRIBUTIONS 

REFERENCE 

BEST  ESTIMATE 

VARIANCE 

(PSEUDO) 

(PSEUDO) 

INDEX 

OF  R 

OF  R 

SUCCESSES 

FAILURES 

1 

0.7273 

0.0145 

7.00 

2.00 

2 

0.S571 

0.0153 

5.00 

0.00 

0.S4SS  0.0073  13.00  1.90 
0.0100  0.0320  4.02  2.71 
0.0100  0.0220  4.02  2.71 
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Exhibit  20 


OUTPUT  FROM  MODE  0 


(Continued) 


THE  CURRENT  RISK  IS  1433. 


COMPONENT 

NUMBER 

SYSTEM  RISK 
IP  SUCCESS 

SYSTEM  RISK 

IF  FAILURE 

EXPECTED  SYSTEM 
RISK  IF  TESTED 

CHANCE  FROM 
CURRENT  RISK 

COST  (*>  PER  UNIT 
CHANCE  IN  RISK 

1 

1444. 2S 

1926.37 

ISIS. IS 

137.74 

1  oBl 

3 

1371.82 

1434. as 

1404.93 

47.94 

7.30 

I  COMPONENT"  3  SUCCESSES"  0. 

FAILURES-  0. 

THE  CURRENT  RISK 

IS  1433. 

COMPONENT 

SYSTEM  RISK 

SYSTEM  RISK 

EXPECTED  SYSTEM 

CHANCE  FROM 

COST  (»>  PER  UNIT 

NUMBER 

IF  SUCCESS 

IF  FAILURE 

RISK  IF  TESTED 

CURRENT  RISK 

CHANCE  IN  RISK 

1 

1444. 3S 

192S.37 

1313.13 

137.74 

1.B1 

3 

1371. 82 

1434. BS 

1404.93 

47.94 

7.30 

COMPONENT-  1  SUCCESSES"  7. 

FAILURES-  0. 

THE  CURRENT  RISK 

IS  933. 

COMPONENT 

SYSTEM  RISK 

SYSTEM  RISK 

EXPECTED  SYSTEM 

CHANCE  FROM 

COST  (•>  PER  UNIT 

NUMBER 

IF  SUCCESS 

IF  FAILURE 

RISK  IF  TESTED 

CURRENT  RI8K 

CHANCE  IN  RISK 

1 

923.74 

1077.03 

933. 7S 

21.22 

11. 76 

3 

79S.44 

1043.44 

902.30 

32.42 

4.4S 

COMPONENT"  3  SUCCESSES-  B. 

FAILURES"  1. 

THE  CURRENT  RISK 

IS  304. 

COMPONENT 

SYSTEM  RISK 

SYSTEM  RISK 

EXPECTED  SYSTEM 

CHANCE  FROM 

COST  (•>  PER  UNIT 

NUMBER 

IF  SUCCESS 

IF  FAILURE 

RISK  IF  TESTED 

CURRENT  RISK 

CHANGE  IN  RISK 

1 

43S.23 

734.13 

477.94 

23.79 

9.49 

3 

400. 97 

327. BS 

490. 04 

4.S7 

71.90 

ESTIMATES  Of  RELIABILITY  DERIVED  FROM  POSTERIOR  DISTRIBUTION* 


REFERENCE 

BEST  ESTIMATE 

VARIANCE 

(PSEUDO) 

(PSEUDO) 

INDEX 

OF  R 

OF  R 

SUCCESSES 

FAILURES 

1 

0.0404 

,  0.0030 

17. B2 

1 .03 

2 

0.9204 

0.0044 

12.00 

0.00 

3 

0.9334 

0.0013 

40.00 

1.03 

4 

0.7430 

0.0097 

12.03 

3.71 

3 

0.7430 

0.0097 

12.02 

3.71 
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LISTING  OF  PROGRAM  CODE  OF  ABRAM 


DOUBLE  'RLLISIOn  1  .  K  .  .  A .  D 

common  ri  a oo  >  .r:i  ioo>.  ic <iov >  .icn<  ioo)  .  icost  < ioo> . 

Umax.  icn.v« 

NODE-O.  TllCN  TES'  RCSUs’S  EMIEPLD  EE  l)UC  NT  1  ALL  Y 
M0DC*1.  I  YLN  1EE1  RESULTS  LN1ENLD  SIMULTANEOUSLY 
kLAI'IU.2)  MODE 
FORMAT  <  11  ) 

BO  2J-i  1JN*1.100 
)l  ( 1  jr.  >«o. 

I;  2<  1  jM*i' . 

11AC-0 
CAL-  I  ttli  AT  A 
Wl;I  IE<«.->?I 

f  OINAT<  •  1  '  .  T  13,  'AUTOMATED  DAYESIAN  RELI ADIL2  T  Y  ASSESSMENT', 

1'  MODEL '. IX. T43, ' -  --  -  -  . .  . . . 

2  '  - V/IA,  IS?,  'EI'lTEl'  INPUT  I* A T A  ‘  / 

2  ix. ts?, -  - 

URIICIV,?) 

l  UKMAI  (  IX,  (JO,  Ktr  EI.CNCC  •  > 

i  iso.  coni  iuukation  component  cost  <*>  or •  / 

1  IX.  MO.  •  INDEX'  ,  lOx.  ■  CODE  '  .  1 1 X  ,  ' NUMBER '. BX ,  '  TESTING'/ 

2  1X.T3U.  '  .  . -',5X. 

3  '  -  •  '/) 

00  11  1-1.1MAX 

wr.i  re  <?,3)i .  ic<  i  > .  1CN(  l  j ,  icost  <  i  > 
r OKMA  T<lx«T41,I3,T33,ll*,)?l,I2,TU3,I3) 

1C< 1 '-10OO0O 

CALL  PI1IURS 

CALL  SYSPRI < I  TAG) 

irdTAO.LO.llCO  TO  200 

CALL  CYSRCL 

URITLIV.UOl) 

UKilC<9.101) 

roRBAT<///.Vl',r3B,  'ESTIMATES  OF  RELIABILITY  BENI  VLB  TKOM 

1  'PRIOR  DISTRIBUTIONS'/) 

FORMAli  /IX. 133. 'REFERENCE  BEST  ESTIMATE', 

2  5X. 'VARIANCE  <*StUDO>  CPSEUBO) '/1X.T35, ' INBEX' , IS 

3  -or  R'.nx.'or  r*  .  ?x.  *  successes  rAiLURCS'/ix. i jj.3< •- 

4  .  —  .  .  . . /) 

DO  AO  I*  1.1  MAX 
EEL 1  *0 

VAR-I<2<1)  Kl<l)«Rl<i> 

n  <var.lt. oesei’I 
ir<LSLI .to. 1)00  10  200 

CALL  BET.KRIC)  ,11211  1  ,A.B) 

Will  TL<'/.  DI.Klll)  ,  VAR,  A,  li 

F0RMA1  <  IX.  1  Jl,  I3.1S1.T  7.4.  I6t,l  7.4,1  70.F0.2, T?2«K7 . 2> 

11 IMOBC.LQ. 1 >00  10  AO 
CALL  DECIDE 


130  10  100 
CALL  I  ESI R 
URilE<V,102) 
rURMAI  t '  I  ' 


•  PI'*'  II  NT, IN  111  S’ PI  Hill  lllfir,  ■ 


'  1 30. 'LSriMATLS  or  reliability  dlriveb  knom 


oi.iu-.'/.ioi) 

BO  ISO  1*1 , I MAX 
VAI.-R2I1 ) -ftl  <  1  )»i:l  <1 ) 

CALL  1-E1  AIR]  <  1 )  ,R2<  1  >  , A.B ' 

OKI  TC<9.1>1,R1  (D.VAR.A.D 

co  ro  300 

01:11  E  <9.301 ) 

FORMAT <//'  LOMRONENI  AND  SYSTEM  PRIORS  ARE  INCONSISTENT ' ) 

STOP 

END 


LUDROUT  I NC  PRIORS 


HAVE  SAMC  EOMfONLNTS  IN  Dll  I  Ll.LNl  PARTS  OT  THE  SYSTEM, 
SHOULD  SI'CCITY  PRIORS  I  Ul:  L'JI!’- 1ST LNCY 


DOODLE  PRECISION  1:1 .1:2.  XI ,  X2 .  A ,  l- 

COMMON  RI  <  10'.  ,P2<100.*  ,1C'  100)  .  ICfl'.  100)  .  ICOST  <100)  . 
1IMAX.1CMAX 
f:l<l)-l./2 
f:2<  1  )*I .  /3 
RC  AD<  8.1)  N'JMPRl 
rOPMATl 12) 


N 

LISTING  OF  PROGRAM  CODE  OF  ABRAM  (Continued) 

68. 

If  INUKffrl  .EQ.OJCU  TO  10 

89. 

C  CC.nE  CCinrOHCNT  HUCT  HAVE  '.AnL  '  !:iuf 

90. 

c  c 

rSILrt  Or  SL'fcSYJTTCft  T£lT  RESULTS  AKL  LOrtl'JNEI'  WITH  COMPONENT 

91  . 

c 

TEST  kCSULTS  l!  INf'JlIlK.  I Hi.  f  0  (H5I.  ft:3  5 1ST  Lit  Ck 

92 . 

c 

SUi'CY'J 7  £fl  PRIORS 

92 . 

WRITE  <9»36> 

94. 

UF:1  TE  (  9f  101  > 

95. 

I«0  10  1  *  1  f  NUrtF’Rl 

96. 

KCAI*<B»2>IC1  rlA.16 

97. 

2 

1  OrnATC15»2I3> 

96 . 

xrtici .eq.o) ic:»iooooo 

99. 

1*0  20  J*1 1  IflAX 

00. 

IFUCI.EO.ICC  J>  )G0  TO  25 

01 . 

20 

continue 

02. 

25 

A*lA 

03. 

04. 

CALL  MOUNT  <A»I*»X1  »X2> 

05. 

Y»X2-X1*X1 

06. 

IF ( Y . LE . 0 . )G0  TO  360 

07. 

WRIT  E 9 »  35  >  J  *  XI »  Y » A  >  i< 

08. 

HI  <  J)*X1 

09. 

f:2  <  j  .*  «X2 

10. 

10 

CONTINUE 

11 . 

101 

FORMAT <  /1XfT33» 'REFERENCE  I'EST  ESTIMATE '  f 

12. 

2  5Xf  'VARIANCE  1PC£UI*0>  <  ILCUI'O  > ' /IX  r  I  35 »  '  INDEX  '  .  T52* 

13. 

3  'OF  fc'»UX»'9r  R'#7X.  SUE  ’-LOSES  F  AI  LUKES  '  .'IX »  T33  ?  3  <  ' - 

14. 

15. 

36 

FQF.'MAT  ( /////  '  2  '  f  T  46  :  '  LF'ECJF  ICATI  On  OF  PRIOR  INSTRUCTIONS  '» 

16. 

1  '  ANI'/QF:'//1X»  1S2»  '  SYLTLM.  E'JI'StStCM  TEST  RESULTS'/) 

17. 

35 

Forrn/IT  <  1 X  ,  T36 . 1 3 .  T!  1  .  r  7 . 4  ■  !  6£  r  r? .  .  .  7  ?U .  re  .  2.  T92.  F7 . 2  .• 

18. 

360 

R  TURN 

19. 

ENO 

20. 

21. 

2?  - 

c 

23. 

SUIT:0UT1NC  INl'ATA 

24. 

c 

25 » 

1‘OUt'LE  PRECISION  Rl»R2 

26. 

CunrON  f'lll'.>0.'.K2<lOV  rIC<l00>.ICN'100.'.lCOSTU00). 

27 . 

1  I  max • I cmax 

2B. 

REAI»<  8»  1  >  MAX  •  T f.HAY 

29. 

1*0  10  1  *■  1  *  MAX 

30. 

r<LAl*<0*2>  I L'  1  .»  *IC?P  2  ;  .1EU51  <  1  > 

31. 

10 

CONTINUE 

32. 

I MAXIMA* 

33. 

1*0  100  1*1  r  100 

3.. 

IT'l  .GT  .  IMAX.'GO  70  150 

35. 

1»U  75  J*1  f  5 

36. 

N-ior?j 

3?. 

M»lo*r ( J  1 ) 

38. 

11*1  Cl  F-MUIKICCIMNJ/M 

39. 

IF 'iriCIT.CT. 1)00  TO  100 

40. 

ir  <1I*1GIT.NC.  1  )G0  TO  75 

41  . 

NCHCR»< IC< I )/N)*N 

42. 

00  50  f>2  » in AX 

.3. 

IF(1C(R)  .  CO .  NCHCK  >G0  TO  100 

44. 

50 

CONI INUC 

45. 

lflAX-InAX  +  1 

46. 

2  C  ( I  MAX  )  ■’NCHCK 

ICN'lMAX>*0 

.8. 

IC05T  < IMAX)»0 

49. 

GO  TO  100 

50. 

75 

CONTINUE 

51. 

100 

CONTINUE 

52. 

150 

CONTINUE 

S3. 

I'O  250  I'l.lrtAX 

54. 

DO  200  J-J»IMAX 

55. 

1F<1CCI).LC.IC(J))C0  70  200 

56. 

2  CM* I C  < 1 ) 

57. 

ICU)*IC<  J) 

58. 

IC< Jl-ICM 

59. 

ICNH*JCN( 1 ) 

60. 

ICN( 1 )*ICN( J) 

61. 

ICN ( J) *ICNH 

62. 

1 COS TM*J COST <  2  > 

63. 

1C05T(I )-ICOST( J) 

64. 

ICOST(J)»ICOSTM 

65. 

200 

CONTINUE 

66. 

250 

CONTINUE 

J 
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LISTING  OF  PROGRAM  CODE  OF  ABRAM  (Continued) 


2i*7. 

16B. 

170. 
171  . 

1 

C 

rORflAT  (213) 
rOfcttAT  (  25,12.13) 

RETURN 

CNI* 

173. 

C 

1  •«. 

SLTROuriNC  SYSRF.l  ( 1TAG) 

175. 

C 

176. 

DOUDlE  PRECISION  R1  .F:2.  ROlD>R:OlD>21 . 22-23. 2. 

177. 

COMMON  R1  (100)  .F:2t  11)0 .IO  100?  .  1CM<  100)  <  I  COST  ( 100)  . 

176. 

1  IflAX.ICMAX 

179. 

DIMENSION  LI  ( 10)  »L2<  10)  .L2'.  10)  .L.'  10)  »L5<10> 

180. 

C 

181 . 

C  COUNTS  NUMDER  or  SERIAL  SVDSYSTLMS.  RECORDS  INDEX  FOR  FL'TURE 

182. 

c 

NX*  THE  MUHDEF:  or  SYSTEM  IN  THE  XTH  LEVEL 

183. 

c 

LX*  THE  LOCATION  OF  THE  XTH  DllilT  STSTEM 

184. 

c 

IX*  THE  XTH  SYSTEM  DClNfa  CONSIDERED 

185. 

c 

186. 

ITAG-0 

187. 

N10-0 

188. 

N1*0 

109. 

IL-2 

290. 

lu-lnAX 

191. 

no  10  j«iLtiu 

192. 

ir(hOU(IC(I) *10000) .NE. 0)00  To  10 

193. 

N1 »N1 + 1 

194. 

LI CN1 )•! 

IVi. 

L  lAl.Cw  UU)  UNLLR‘1  A1H1  T .  AN1'  LUUtl!2  HJflfcLP  If  0VLKF1  IH.5 

196. 

11  <RI < 1 • ,L0.0)00  TU  10 

197. 

K'l  <  1 ) *K*1  <  1  > /f:i  i  1 ) 

196. 

f:2 1  i  >  *R2  ( i  >  /r<2  <  i  > 

199. 

innill)  .  GT  .1  )G0  TO  105 

200. 

ir<R2(l).GT.l)C0  TO  105 

201. 

N10-N10F 1 

202. 

10 

CONTINUE 

203. 

N100-N1-N10 

20.. 

C  SET  UPfER  LIMIT  ON  INDEX 

205. 

LICNl.D-lHAX.l 

206. 

c 

20?  ■ 

ir<Nl .EO.O)GO  TO  101 

208, 

I'D  100  I1-1.N1 

209. 

1 1 1 "L 1  111) 

210. 

1DR1<I11).NE.0)G0  TO  IS 

211. 

C  ALLOCATES  REMAINING  UNCERTAINTY 

212. 

R1(II1  )»r:l  (1  >?»<!  ,/NlOO) 

213. 

R2  < 1 1 1 )  »K2  <  1  > »» ( 1 .  /N1 00 ) 

214. 

c 

215. 

IS 

N20*0 

216. 

N2‘0 

217. 

jL>Li(imi 

210. 

JU-L1  (ll-tl)-l 

219. 

1JCHEK-I1 

220. 

If(  JU.LT.  JDCO  70  100 

221. 

c  count  cr< 

•»'>*» 

I'O  20  J-JL.JU 

223. 

ir  <IC<  J1/10000.ME.  IJCHCt.)  GO  TO  20 

224. 

IF  (MOD(  lC(J)flOOO) . NE . 0 ) CO  TO  20 

223. 

NC-’Na-fl 

226. 

L2<N2)*J 

227. 

ITlFTK  J)  .  EO.O.'GU  TO  20 

228. 

R0LD-R1 (111) 

229. 

R1  ( 1 1 1  )  -1 .  -  <  1 .  -  ROLD )  /  <  1 .  -f:l  ( J )  ) 

230. 

f:2<  1 1 1  >»<  1 .  -2»I<0LDTR2<111))/(1.-2,R1(J)+R2<J)>F2*R1UI1> 

231. 

1F(R1<II1).GT.1.)C0  TO  105 

1F(R2(II1).GT.1.)G0  TO  105 

233. 

N20-N2041 

234. 

C  COUNTS  NUMDER  or  overt: HUNG 

235. 

20 

continue 

236. 

N200»M2-N20 

237. 

L  2  ( N24 1  )*JU-»  1 

236. 

c 

239. 

IF <N2.CP.0>G0  TO  100 

240. 

DO  200  12*1 »N2 

241. 

1I2*L2( 12) 

242. 

:r<Rl<lI2>  .NE.OGO  TO  25 

243. 

C  DON'T  DOTHER  WITH  ITEMS  THAT  ALREADY  HAVE  A  RELIAD1L1TY 

244. 

21* ( 1 . -R1 < 1 1 1 >  )T» l : .  'N209 ) 

2.5. 

22* ( 1 .  -2,R1  ( III  >41:2(111  -  ■»»'.  1  ,.'N290) 

2.6. 

HI < 1 12) *1 . -21 

247. 

R2<  II2>»22*2*F:l  (122)-1. 

%  ^ 
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VI. 

LISTING  OF  PROGRAM  CODE  OF  ABRAM  (Continued) 

248. 

c 

249. 

'll 

N30-*0 

250. 

N3*0 

251  . 

KL«L2< 12)41 

252  T 

KU«L2(  I2U>-1 

253. 

IKCHEK-IirlO+12 

254. 

1F(KL.GT .KU>G0  TO  200 

*trr 

P0  30  K-F.L»KU 

256. 

IF  <  2C(K>/1000 .  NE  .  IF.CHEK1G0  TO  30 

257. 

IF(M0B(IC(K>.100J  .NE.O.'GO  TO  30 

258. 

N3-N3+1 

25  V . 

L3 ( N3  >  »K 

260. 

iF(fci <k  > .pn.o>nn  to  30 

262  . 

i:i<U2)«r:a  (112)/kl 

262. 

1 :2  ( 11 2 )  »f;2  <  1 1 2 )  /R2 t K ) 

263. 

IT (Ri <122). Cl .1. ) CO  IQ  105 

264  . 

II  <R2< 112) .G1 .1 . )G0  TO  105 

265. 

N30-N3041 

266, 

30 

CONTINUE 

267. 

N300*N3-N30 

266 . 

L3(N34  1  >»MJ+1 

269. 

C 

270. 

IF<N3.EQ.9)C0  10  200 

271  . 

DO  300  I3«=l  » N3 

'172 

1I3»L3(13) 

273. 

IF  <F:i  (113)  .NC.OGO  TO  35 

274. 

K1 < 1I3)»R1 <112)99(1 ./N300) 

275. 

R2<II3)*R2< XX2)9*<1./N300> 

276. 

c 

277. 

35 

N40-0 

278. 

N  4*0 

279. 

LL  «L3 ( 13)41 

260 . 

LU«L3<I3T1)-1 

281. 

ILCHEf.«l  1.1001  I2»10<13 

282  • 

IF<LL.GT.LU)GO  TO  300 

283. 

GO  ,0  L-LL'LU 

284. 

1F(IC(L)/100.NC  .  ItCHLr. >C0  TO  .0 

285. 

ir(«oixic(L).io).HC.o.'Co  to  ,0 

286. 

N4=N441 

287. 

L4<N4>-L 

208. 

IF ( Rl < L ) . CO . 0 ) GO  TO  40 

289. 

N40*N404 1 

290. 

K10LD*r:l  (213) 

291. 

Rl  <1I3>*1 .  -  <1 .  -RIOlI')  / <  1  •  -Rl  <L)  ) 

292. 

r:2ai3)*<l.-2»R10LD<R2(H3)  >/<l  . -2»F:l  <L)4F:2<L) ) 

293. 

H2»R1<II3>-1 . 

294. 

IF<R1<II3).CT.1.)C0  TO  1C5 

295. 

IF(R0(II3) .CT.l . >G0  TO  105 

296. 

40 

CONTINUE 

297. 

N400*N4-N40 

298. 

L4<N441)-CU41 

299. 

C 

300. 

1F(N4.ECI.0>G0  TO  300 

301. 

DO  400  1 4«1 » N4 

302. 

IJ4-L4U4) 

303. 

IF<R1(1I4) .NE.O>GQ  TO  45 

30.. 

23«(1. -Rl (II3>  >»»(1  ..'K400) 

305. 

24"  C 1 .  -  29F:1  <113)4  f :2 1 1 1 2  5  >  9?  <  1 . 7N400 ) 

306. 

Rl (1I4)«1 .  23 

307. 

R2(  1 1 4  > -Z4  <  2 .  *f:l  ( 1 1 4  >  - 1 . 

308. 

C 

309. 

45 

NSO-O 

310. 

N5*0 

311. 

f1L»L4  (14)41 

312. 

MU»L4(1442 ) ~ 1 

313. 

1HCHEK-I 1910004 129 1 00  f  139104  14 

314. 

if<ml.gt.m'.ugo  to  400 

315. 

DO  50  N*KL>MU 

316. 

ir<IC<N)/lO.NC.lML*!<EK'GO  TO  50 

317. 

ir<H01KlC<fl>  *10)  .£0.0)00  TO  50 

318. 

N5»N541 

319. 

LS(N5)»h 

320. 

1F(R1 (h) .  EG . 0 ) GO  TO  50 

321. 

N50*N50T 1 

322. 

Rl < 1 14 ) *Rl (114) /Rl <  H) 

323. 

R2< 1 14  >  »R2 ( 3 1 4  J /R2 ( H  > 

324. 

IF <R1(II4) .CT.l. )G0  TO  105 

225. 

IF  <F:2(  II4>  .CT  .1  .  >  GO  TO  105 

v _ 

326. 

50 

CONTINUE 

_ y 

MAXIMUS 


LISTING  OF  PROGRAM  CODE  OF  ABRAM  (Continued)  | 


i  J  7 . 

N5u t*Nt  NSC* 

320. 

L5< N5H 1 ) *HLH  1 

i*»c» , 

IF 'N5.CQ. 0>G0  TO  400 

230. 

DO  500  I5«1»N5 

221  • 

1 15*l_5<  15) 

2 32. 

ir tKl ( 11Z> .NE. 0)DU  To  500 

333. 

Rl<115>*Rl (I14>»*(1 ./N500) 

234. 

R2  <  1 1 5  >  «F:2  ( 1 1  4  )  **  ( 1 .  /N500  ) 

335. 

500 

CONTINUE 

334. 

400 

CONTINUE 

227. 

300 

continue 

230. 

200 

CONTINUC 

239. 

100 

CONTINUE 

340. 

CO  TO  101 

341  . 

105 

1TAG-2 

342. 

101 

RETURN 

343. 

1000 

r  OI.-NAT  U 1 0  •  2X » f  2  0 . 5  f  2X 1  f  1 0 . 5  > 

344. 

END 

245. 

C 

346. 

c - 

—  •  -  -  —  -  -  -  - —  — 

347. 

c 

240. 

SUBROUTINE  CYSREL 

249. 

c 

350. 

DOUBLE  PRECISION  Rl  » 1:2  * I'tlULT  1  »  HHULT2 

251. 

CONflOH  PI  (  1 00 >  t  R2  ( 1 00 )  f  U  (  2  00  >  1 I CN  < 1 00  >  .  I  COST  (100)* 

25*1 « 

UMAX  *  JCflAX 

353. 

DIMENSION  L(200) 

354. 

DO  1234  LL-lflOO 

255  • 

1234 

L ( LL ) *0 

354. 

DO  100  I COUNT  »1  *  3 

25? . 

N*1  C’t  * ( 1  COUNT  - 1 > 

350 . 

I T  AG*1 

359. 

IT  <  I  COUNT  .  EQ  .  2  •  OF: .  I  COUNT  .LQ.  4  J 1 TACcO 

340. 

DO  100  J*-l»lhAX 

341. 

IF ( L  <  J) « EQ . 1 ) GD  TO  100 

343. 

IA«riOD'.  IC(  J)  »100?N)/N 

343. 

H'*non<iA.iO) 

344. 

ir<lA.EO.O>GO  TO  100 

345. 

ir<:n.NE.o>Go  to  so 

366. 

JS-J 

34?. 

I'rtULT  1*1.0 

348. 

DMULT2*1. 

369. 

GO  TO  100 

370. 

50 

IFdTAG.EO.  DGO  TO  75 

371. 

DrtUUT  1  *I*MUL  Tl  *  <  1  -F:i J:  1 

3^3. 

DnULT2-DMULT2»(l  2*R2  t  J  MR2'  J*  > 

373. 

f:l  ( JS>*1  -DflULTl 

374 . 

R2( JS)*DMULT2l2. *R1 ( JC > -  1 . 

375. 

4< J)»l 

37«. 

GO  TO  100 

377. 

75 

I'«ULT1»I'ML'LT1»F:1  (  J.' 

378. 

Ei.m  UL  T  2  r  DM  'JL  T  2*F:2  <J> 

379 . 

feu  JS>» DMULTl 

380. 

R2( JS1-DMULT2 

301. 

L< J)«i 

382. 

100 

CONTINUE 

383. 

return 

384. 

CNO 

305. 
XftA  . 

c 

3B7. 

388 . 

389. 

390. 

391. 

392 . 
J93. 
J94. 
395. 
394. 
39?. 
390. 

399. 

400. 
401  . 

402. 

403. 

404. 


SUBROUTINE  1C Cl DC 

D0UPLE  PRECISION  Pi  »f':2f  R2H*P2K»X1  »X2» A*D » APLUSl  » 

1  HI  1  *  R1 2 * DPLUS2  »X3*X4*F:2l  *R22 
rrmnnw  R’l  .irncr  MftAu 

Umax*  IC«ax 

1>I  AENCI0N  R1H(2'J)  #R2H<  20) 

CRlCh*<R2<  1  J-Rl  <  1  /•*l*)»U-0O00 
WRITE < 9*222 > CP ISP 

rOf:«AT<//'l  '  »29H  THE  CURKLNT  PIGf. 

WHITE <9*11  ) 

T  OPttAT  </41X*Tl?r  *  COMPONENT  STSICh  KlCf 

1  •  L XI  CL  r LI*  gyctem  change  r  POM 

2T19. 'NUMPLP  If  CUCCLS.  If  FAILURE 

3'  CURRENT  PJCf.  CttftNLC  IN  RIGf  '  '1X*T17 

4  IXr  '  ....  . - 

5 '  -  -  -  •  -  '  '  -  '■'> 


s*2x.rio.o> 

crciEfi  Pic 
COST  '.!>  PU¬ 
RIST.  jr 


h  * » 

unit  * /IX. 
ICCTCl'-  I 


J 
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VI.  LISTING  OF  PROGRAM  CODE  OF  ABRAM  (Continued] 


405. 

I'U  100  1*1  •  ISriA* 

•406. 

K-0 

4Q7. 

HO  30  J* 1,1 MAX 

40U. 

ll  <  JCNt  J)  .WL.  I  .'CO  10  30 

4  Or. 

L  COUNTS  NUfll'Ll:  Of  lltLNlICAL  COflf UNtlNTS  ANl*  STOKES 

410. 

1 

411  . 

r:iH<r.)*r:i  t  j> 

412. 

Kju(h)*r:2<  j> 

412. 

30 

CUNTINUE 

414. 

C 

415. 

CALL  EiETA<r:lH<  1  >  ,I  2H<  1  )  ,  A.t) 

416. 

ARLUSl-Al  1 

417. 

CALL  MOMN1  (AfLOCl  .fc.Xl.X2> 

410. 

no  40  j*2»i«r,x 

419. 

IF  UCN'J) .  N£  .  I >CO  10  40 

420. 

ftl ( J)-X1 

421  . 

R2< J)»X2 

422. 

40 

continue 

423. 

C 

424. 

CALL  SYSREL 

425. 

Rll-Rl (1 ) 

426. 

R12*R2U  > 

427. 

E»RLUS1»I«*1 

428. 

CALL  MUrtNT  (A.1PLUS1  f  X2»X4.' 

429. 

no  60  J*1 » I  MAX 

430. 

IF  < ICN( J) . NC . I )GO  TO  60 

431. 

F:l  {  J)»X3 

432. 

T:2<  J)*X4 

433. 

C  JH» 

LAST  INI'EX 

434. 

JH*J 

435. 

60 

continue 

436. 

C 

437. 

CALL  SYSREL 

430. 

F:21*F:l  <  1 ) 

429. 

R22*F:2  ( 1 ) 

440. 

F:iSM*'ru2-r:i:»»2>»ieoooo 

441, 

RISKS* <K22-R21»»2>»1COOOO 

442. 

CRIUK.RIHU  >»RISKH  (1  R1K<  1  >  ?, RISKS 

443. 

I«RISh*CRISr-ER2SK 

444. 

ECQST-ICOST<  JHI/I'RISK 

,«s. 

K-0 

446. 

I'O  00  J*1  »  I  MAX 

447. 

IF'ICN'J'.HS.DCO  TC  60 

448. 

T.»K*1 

449. 

Rl<  j)*F:iH(h> 

450. 

F:2  ( J )  *F:2H  < K  i 

451. 

80 

CUNTINUE 

452. 

URl  TE ( 9. 1 > I . RI SKI »  RISKS. EKISK . BRISK  .  ECOST 

453. 

1 

FORMAT  < IX , T20 . 1 3 . T 2 3 . F E . 2 . T,9 . F8 . 2 . TA7 . TB . 2 . ' 

454. 

100 

CONTINUE 

455. 

REAP  <8*2)1  CONN  r  2  A »  Z  8 

456. 

XS*IA 

457. 

xr*ip 

4MA. 

TFnrnHN.ro.rt>Rn  to 

45V. 

C  LCHU  I  COMIUNCNT  CL'LCLLi  AN1  FAILURES 

460. 

uruu  v»5;iL0nNrXC.  xr 

461  . 

«/ 

r  URMAI  (  /  *  1 '  .  1  OX »  'EOMIUNENT*  •  .  I3.3X.  '  SL'CLEECE! 

462. 

1  ' T A 2 LURES* ’ $f  3.0//) 

463. 

CALL  UPDATE  aC0rtNrX5»XD 

464. 

CALL  SYSREL 

465. 

CO  TO  50 

466. 

*» 

FORMAT  <  22’2I3> 

,*7. 

120 

CALL  SYSREL 

460. 

RETURN 

469. 

CNH 

470. 

C 

471. 

c - 

472. 

c 

472. 

CL'I’RDUI  INC  ur-IcATL  !  ICOMN.XC.  XF 

474. 

c 

475. 

DOUPLC  PRECISION  Rl » R2 » A » L 

COMMON  Rl < 1 00 > . R2 1 100 ’ . 1 C < 1 90 • . I CN ' 1 00 ) . 1 COS 

477. 

1 1  MAX  t ICttAX 

,70. 

I'O  100  2*1 »  lflAX 

479. 

irUCN'I)  .NC.IS'JMNJSP  TO  ICO 

480. 

CALL  i'E  f  A  <  l\’l  <  I )  »  R2  <  I  >  » A » H  ) 

481 . 

A*  A  f XS 

482  • 

n*nxr 

483. 

CALL  MOMNT  ( A.I’.Rl  l  J  >  .K2!  1  >  ' 

404. 

100 

CONTINUE 

485. 

RE  T URN 

486. 

END 

» F0 . 2 f T105 


•T3.0f 2X» 


t F8 .2 ) 


J 


MAXIMUS 


VI. 


LISTING  OF  PROGRAM  CODE  OF  ABRAM  (Continued) 


4Z7. 

486. 

489. 

490. 
493  . 
491*  • 

493. 

494 . 

495. 
496  • 

497. 

498. 

499. 

500. 
5C1 . 

502. 

503. 

504. 

505. 

506. 

507. 
500. 
509. 
*20 . 
531  . 

512. 

513. 

514. 

c* 


C 

c 

SUBROUTINE  7CS1F: 

C 

double  precision 

common  ni  c  l oo ) » f:2 a oo ; » i c <  i oo ?  *  i cn '  i oo>  •  i cost  < i oo  j  » 

1IMAX.ICMAX 
f<CAD<0»  1  3NUMC0M 

1  FORMAT  < 12) 

UF.I  T£  ( 9r  51 ) 

51  F  OF.MAT  (////.'/ '1  '  rT55»  COMPONENT  TEST  RESULT S  '  , /// 

1  IX » 7  48»  'COMPONENT  SUCCESSES  FAILURES'/ 

2  IX,  T4Br  ' - -  - -  - '//; 

DO  10  1*1  *  NUMCQM 

F.'CAD ( 8 » 2 )  1 COMN ,  1 A ,  ID 

XC*  1 A 

XF*ID 

UK  I T£ ( 9 , 50 ) 1 COMN . XS  r  XF 
50  1  0f:MAT(lX»T51t  I3r  T65»r3.0rT79»F3.0> 

2  FORMAT  U2»2I3> 

CALL  UPDATE  '  ICOMR*  X£ ,  XF  ) 

10  CONTINUE 

CALL  SYSREL 
RETURN 
END 
C 

c- . - . . . .  - . . 

c 


514. 

51?. 

516. 

519. 

520. 
£21 . 

522. 

523. 

C 

C 

SUDROUTI NE  BETA '  F:l » R2 »  A  r  B 
DOUBLE  PRECISION  F.'l  t  R2 » A r  2 

A*  ( r:i  *ri  -  ri  *R2 >  / t  r;2 -r.itr:  1  >  - 1 
B*<1 .  -  R1  )*  (A-l  1 .  )/Rl  -1 . 

RETURN 

END 

525. 

C 

526. 

SUPROUTINL  NlMHMA-l>»Rl»R2> 

527. 

DOUBLE  TRLCISION  RlrR2»A»D 

520. 

C 

527. 

R1MAI1  >/<AU»l2.  ' 

530. 

R2*R1»<AI2.  >/<All't3. ) 

531. 

R Cl  URN 

532. 

END 

MAXIMUS 
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PART  4 

SURVEY  AND  TESTING  ORGANIZATIONS  IN  DOD 


In  this  part  of  the  report,  we  present  a  directory  of  Service  organiza¬ 
tions  involved  in  reliability  testing,  and  the  results  of  our  survey  of 
these  organizations. 

I.  DIRECTORY  OF  ORGANIZATIONS  SURVEYED 


The  purpose  of  this  section  is  to  present  an  overview  of  organizations 
in  each  of  the  Services  involved  in  reliability  assessment.  Generally, 
there  are  five  major  activities  in  reliability  assessment  in  which  these 
organizations  are  involved. 

•  Research  of  new  methods  of  assessing  reliability  and  establishing 
requirements  for  new  systems; 

•  Development  of  new  systems  or  items  of  equipment  from  initial 
design  through  contract  award,  prototype  development,  and  pre- 
production  testing; 

•  Field  Testing  systems  at  various  development  phases,  prior  to 
final  production,  and  at  intervals  during  in-service  lifetime; 

•  Logistical  Support  of  systems,  providing  a  supply  of  spares, 
maintenance,  and  assuring  the  operational  readiness  of  systems;  and 

•  Reliability,  Maintainability,  and  Quality  Assurance  policy  guid¬ 
ance  development. 

A  Program  Manager  is  usually  assigned  the  primary  responsibility 
for  development  of  a  particular  system.  Depending  on  factors  such  as 
the  cost  and  importance  of  the  system,  an  individual  or  group  of  indivi¬ 
duals  is  assigned  responsibility  for  reliability  growth  and  assessment. 


MAXIMUS 


Expertise  in  all  the  above-mentioned  areas  is  expected  to  be  used  by 
the  Program  Manager  during  the  development  phase.  A  description 
of  the  manner  in  which  the  responsibilities  for  reliability  assessment 
are  carried  out  in  each  of  the  Services  follows. 


A.  U-  S.  ARMY 


Reliability  assessment  activities  are  carried  out  under  the  purview 
of  three  of  the  five  Deputy  Chiefs  of  the  Army,  as  shown  schematically  in 
Exhibit  1  on  the  following  page. 

•  Deputy  Chief  of  Staff  for  Research  Development  and  Acquisition 


(DCSRDA)  -  generally  responsible  for  developing  future  improve¬ 
ments  and  advising  on  scientific  and  evaluation  issues.  The 
Operational  Test  and  Evaluation  Agency  (OTEA)  reports  to 
DCSRDA  as  do  other  field  and  staff  support  agencies  involved 
in  this  type  of  activity. 

•  Deputy  Chief  of  Staff  for  Logistics  (DCSLOG)  -  responsible  for 


operational  availability,  control  of  inventories,  and  the  manage¬ 
ment  of  weapon  system  and  equipment  development.  The  Material 
Development  and  Readiness  Command  (DARCOM)  is  under  the 
staff  purview  of  DCSLOG. 

•  Deputy  Chief  of  Staff  for  Operations  and  Plans  (DCSOPS)  -  has 


purview  over  two  commands:  Forces  Command  (FORSCOM)  and 
Training  and  Doctrine  Command  (TRADOC).  The  former  is 
responsible  for  field  strength  and  operations.  The  latter  command 
supervises  CONUS  Army  schools  involved  in  training  activities 
and  doctrine  development,  MASSTER,  the  major  field  test  opera¬ 
tion  at  Fort  Hood,  Texas,  and  the  Combats  Development  Command 
(CDC). 

Reliability  contracts  within  these  agencies  and  commands  are  given  below 

along  with  brief  descriptions  of  their  interaction  in  the  reliability  growth 


and  assessment  processes. 


[i; 


»  "  V 
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Exhibit  1 

U.  S.  ARMY  ORGANIZATIONS  INVOLVED  IN 


RELIABILITY  TESTING  AND  ASSESSMENT 


DCSOPS 


DC SLOG 


TEA 


FORSCOM 


TRADOC 


DARCOM 


•TCATA  (MASSTER) 

Combat  Developments 
Command 


■Readiness  Commands 


■Systems  Commands 


TECOM 


■  R  &  D  Commands 


•Troop  Support  Command 
■Harry  Diamond  Labs 


v  '* 
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l.  Development  &:  Acquisition 


The  development  and  acquisition  of  major  systems  is  coordinated  by 


twenty-two  project  managers  within  DAR COM's  Office  of  Project  Manage¬ 


ment.  Procurement  and  development  of  components  and  subsystems  are 


carried  out  by  product  managers  within  the  various  agencies  and  commands. 


Certain  high-cost  systems,  however,  such  as  theXMl  tank  and  the  Advanced 


Helicopter  are  handled  by  program  managers  at  the  DOD  level.  These 


DARCOM  project  managers  are  autonomous  in  that  they  report  directly  to 


the  Secretary  of  the  Army  when  required.  Project  Managers  prescribe 


their  own  reliability  assessment  plans  under  the  command's  general  poli- 


DARCOM  is  responsible  for  logistical  support  as  well  as  acquisition 


and  development.  The  various  commands  within  DARCOM  are  split  into 


readiness  (logistics)  and  development  oriented  commands.  Reliability, 


availability,  and  maintainability  (RAM)  functions  are  considered  part  of 


product  assurance  and  are  overseen  by  the  Reliability  and  System  Assess¬ 


ment  Division  of  the  Directorate  for  Quality  Assurance  (Mr.  A.  Nordstrom 


202/274-8912)  at  DARCOM  Headquarters.  Each  of  the  ten  commands  in 


DARCOM  has  RAM  personnel  involved  in  product  assurance  for  new  mate¬ 


rial  development  and  in  the  analysis  of  data  generated  during  operational 


and  storage  phases  of  item  life  cycles.  A  list  of  RAM  contacts  for  the 


ten  commands  is  given  below. 


SV V  s' V  V  - .  . 


•  V  *  *  .  4  .  %  **.  .  .  \ 
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•  ARRCOM:  Armament  Materiel  Readiness  Command,  Rock 
Island,  Illinois.  Mr.  Robert  McKeague  309/794-4851. 

•  ARRADCOM:  Armament  Research  and  Development 
Command,  Dover,  New  Jersey.  Mr.  Dale  Adams  20 1 /794-667 1 . 

•  AVRADCOM:  Aviation  Research  and  Development  Command, 

St.  Louis,  Missouri.  Mr.  Robert  Neff  314/268-2541. 

•  MIRCOM:  Missile  Materiel  Readiness  Command,  Redstone 
Arsenal.  Mr.  Carl  Coxsey  205/876-5281. 

•  MIRADCOM:  Missile  Research  and  Development  Command, 
Redstone  Arsenal.  Mr.  William  Walker  205/876-7570. 

•  MERADCOM:  Mobility  Equipment  Research  and  Development 
Command.  Fort  Belvoir,  Virginia.  Mr.  Lynwood  Rabon 
703/664-6402. 

•  TAR  COM:  Tank-Automotive  Materiel  Readiness  Command, 
Warren,  Michigan.  Mr.  Edward  Po  lorn  ski  313/264-1100. 

•  TARADCOM:  Tank-Automotive  Research  and  Development 
Command,  Warren,  Michigan.  Mr.  Wilbert  Simkowitz 
313/573-2860. 

•  TSARCOM:  Troop  Support  and  Aviation  Materiel  Readiness 
Command,  St.  Louis,  Missouri.  Mr.  Vail  Miller  314/263- 
2464. 

Other  commands  under  DARCOM  are: 

•  CORAD  COM:  Communications  R&D  Command 

•  ERADCOM:  Electronics  R&D  Command 

•  CERCOM:  Communications  &  Electronics  Command 


•  TECOM:  Test  and  Evaluation  Command 
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In  addition,  DARCOM  manages  several  labs: 

•  Natick  R&D  Cbmmand:  involved  in  testing  of  equipment 
resistance  to  environmental  extremes; 

•  Harry  Diamond  Labs:  involved  in  testing  to  conduct  stress 
and  corrosion  analysis  and  the  impact  of  various  noxious 
environments  on  various  materials  and  other  basic  research 
activities. 

2.  Testing  and  Evaluation 

The  U.  S.  Army  places  heavy  emphasis  on  removing  operational  prob¬ 
lems  prior  to  introduction  of  systems  into  field  use.  As  shown  in  Exhibit  3, 
DCSOPS  has  two  sides:  FORSCOM,  which  is  not  involved  with  reliability 
assessment  or  testing,  and  TRADOC,  the  training  and  doctrine  development 
side,  which  is  responsible  for  ensuring  that  the  new  equipment  or  weapons 
can  be  efficiently  used  by  the  troops.  To  accomplish  its  mission,  TRADOC 
controls  three  primary  activities:  training,  doctrine  and  tactical  development, 
and  field  activities.  Doctrine  and  tactical  development  has  long  been  the 
purview  of  the  Combat  Developments  Command  (CDC),  which  is  now  part 
of  TRADOC.  CDC  interacts  with  the  schools  and  various  forces  and  is 
generally  responsible  for  generating  Qualitative  Material  Requirements 
(QMR's)  to  satisfy  future  needs.  TRADOC' s  field  test  activities  are 
centered  in  project  MASSTER  (the  Modern  Army  Selected  System  Test, 
Evaluation,  and  Review)  at  Fort  Hood,  Texas.  MASSTER,  newly  desig¬ 
nated  as  TRADOC  Combat  Arms  Test  Activity  (TCATA),  brings  together 
groups  involved  through  the  design  and  development  cycle,  regardless  of 


MAXIMUS 


their  organizational  affiliation,  in  its  comprehensive  field  trials  and 
operational  tests.  It  is  at  this  point  that  TRADOC's  System  Managers 
assume  responsibility  for  the  system  from  the  Project  Managers  and 
begin  to  integrate  it  into  supply  depots  and  field  use.  DARCOM  is,  of 
course,  still  responsible  for  the  logistical  support. 

The  White  Sands  test  facility,  used  primarily  by  MICOM,  is  operated 
by  Fort  Bliss  personnel.  It  is  jointly  administered  by  DARCOM  and  TRADOC. 
Ballistic  testing  is  conducted  at  Aberdeen  Proving  Grounds  (APG)  under 
DARCOM  supervision.  However,  TECOM  conducts  most  of  the  APG  testing 
activity  to  validate  or  try-out  new  QMR  concepts. 

Each  of  the  services  has  an  independent  test  agency.  The  Army's  is 
the  Operational  Test  and  Evaluation  Agency  (OTEA)  located  in  Falls  Church, 
Virginia.  OTEA  reports  to  DCSRDA  and  is  responsible  for  certification  of 
the  produced  system  for  field  use.  OTEA  tests  major  systems  and  sub¬ 
systems  in  operational  environments  attempting  to  integrate  tactics,  troops, 
doctrine  support,  etc.  ,  in  realistic  settings.  These  tests  do  not  focus  as 
much  on  reliability  as  on  operational  questions,  such  as  whether  the  troops 
operate  the  hardware.  OTEA  conducts  tests  at  bases  throughout  the  world. 
However,  TCATA  facilities  at  Fort  Hood  are  used  so  frequently  that  they 
maintain  an  office  at  that  base.  The  contact  is:  Mr.  Virgil  Henson,  Chief 
Analyst,  TCATA,  Fort  Hood,  Texas  817/532-9203.  The  OTEA  contact  for 
RAM  assessment  is  Mr.  Frederick  McCoy,  202/756-1028. 


MAXIMUS 


Test  plans  are  usually  developed  by  a  team,  its  members  representing 
the  development  and  production  aspects,  operational  needs,  logistics,  human 
factors,  and  statistics. 

3.  Individuals  Contacted 

The  following  persons  were  contacted  during  the  survey  phase: 

•  Mr.  Arthur  Nordstrom,  Chief,  Reliability  and  System  Assessment 
Division,  DARCOM,  202/274-8912. 

•  Mr.  Larry  Crow,  Armament  Materiel  Systems  Analysis  Activity 
(AMSAA),  Aberdeen  Proving  Grounds,  301/278-3280. 

•  Mr.  Louis  Iannuzzelli,  Armament  Materiel  Readiness  Command, 
309/794-4851. 


•  Mr.  John  Obren,  Director,  Product  Assurance  Directorate, 
Armament  Materiel  Readiness  Command,  309/794-4851. 

•  Mr.  Robert  Launer,  Army  Research  Office,  919/549-0641. 

•  Dr.  Jag  Chandra,  Director,  Army  Research  Office,  919/549-0641. 


B.  U.  S.  AIR  FORCE 

Reliability,  availability,  and  maintainability  policy  is  established 
within  two  of  the  five  Offices  of  the  Deputy  Chiefs  of  Staff  of  the  Air 
Force  as  shown  schematically  in  Exhibit  2  on  the  following  page. 

•  DCS/Research  and  Development  (DCSRD)  -  generally  responsible 
for  development  and  acquisition  of  new  systems;  for  coordinating 
planning  and  program  analysis  and  for  developing  operational 
requirements  of  new  systems. 

•  DCS/Systems  and  Logistics  (DCSSL)  -  generally  responsible  for 
procurement  of  production  approved  systems  and  for  their  logistic 
suppo  rt. 


Exhibit  2 
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The  Office  of  Primary  Responsibility  (OPR)  for  overall  reliability 
policy  for  the  Air  Force  is  Logistics  Engineering  and  Support  Division 
(LGYE).  Reliability  testing  and  analysis  policy  is  coordinated  by  the 
RDXM  within  the  Directorate  of  Planning,  Programming  and  Analysis 
under  the  DCS/RD  in  conjunction  with  the  LGYE  of  the  Directorate  of 
Maintenance  Engineering  and  Supply  under  DCS/SL. 

In  general,  systems  acquisition  and  development  is  directed  by 
program  managers  within  the  implementing  command  which  is  normally 
the  Air  Force  Systems  Command  (AFSC). 

Acquisition  and  system  development  are  coordinated  by  program 
managers  within  the  Air  Force  Systems  Command  (AFSC).  [Development 
and  acquisition  of  most  systems  are  accomplished  by  AFSC  through  an 
AFSC -assigned  Program  Manager  (PM)]  .  The  PM  is  ultimately  respon¬ 
sible  for  all  aspects  of  the  system,  including  achievement  of  RAM  objective 
The  role  of  AFLC  is  that  of  coordinator /monitor.  AFLC  is  responsible 
for  advising  on  logistics  sup  portability  aspects  and  for  planning  for  oper¬ 
ational  logistics  support.  However,  the  decision  authority  remains  with 
the  Program  Manager  (AFSC).  Basic  organizations  within  AFSC  and 
AFLC  are  shown  in  Exhibits  3a  and  3b.  The  Air  Force's  independent  test 
agency  is  located  at  the  Air  Force  Test  and  Evaluation  Center  (AFTEC)  at 


Kirtland  Air  Force  Base.  Reliability  assessment  contracts  within  these 
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agencies  and  commands  are  given  below  along  with  brief  descriptions  of 
their  interaction  in  the  reliability  growth  and  assessment  processes. 

1 .  Development  &  Acquisition  in  AFSC 

Program  Managers  are  located  within  four  major  product  divisions 
of  AFSC  as  shown  in  Exhibit  3a: 


•  Electronic  Systems  Division  at  Hanscom  Field 

•  Aeronautical  Systems  Division  at  Wright-Patterson  Air  Force 
Base 

•  Space  and  Missile  Systems  Organization  headquartered  at  Los 
Angeles  Air  Force  Station 

•  Armament  Development  and  Test  Center  at  Eglin  Air  Force  Base 
The  OPR  for  RAM  for  AFSC  is  Major  Guy  A.  Morgan,  SDDE, 

301/981-3316. 

In  addition  to  the  above  divisions,  AFSC  also  has  several  laboratories 
and  Flight  Test  centers.  The  labs,  with  a  few  exceptions  (such  as  RADC, 
and  the  Flight  Dynamics  Laboratory  at  WPAFB),  do  not  have  reliability- 
assessment  responsibilities.  Although  tests  designed  to  cause  item  fail¬ 
ures  are  conducted,  they  are  usually  for  new  experimental  products  under 
special  conditions  and,  hence,  usually  do  not  generate  data  usable  for 
reliability  assessment. 

During  the  development  of  a  system,  the  contractor  is  responsible 
to  design,  test,  and  make  his  evaluation  of  the  system,  subsystems,  and 
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its  various  components.  In  general,  the  contractor's  test  plans  and 
procedures  are  approved  by  the  Program  Manager  who  arranges  for 
USAF  test  facilities  as  necessary.  Test  and  evaluation  (T  &  E)  Master 
Plans  are  prepared  by  the  responsible  test  organization  (usually  AFSC) 
and  program  management  office.  The  independent  test  and  evaluation 
agency,  AFTEC,  conducts  operational  testing  and  evaluation  (OT  &  E) 
on  early  production  models.  Operational  test  requirements,  established 
by  AFTEC  and/or  the  operating  command,  include  (1)  assessment  of 
operational  capabilities,  (2)  development  of  tactics  and  procedures,  and 
(3)  evaluation  of  logistic  support  capability.  AFTEC  continues  these 
tests  on  production  units  even  after  the  first  units  are  accepted  by  the 
users. 

2.  Testing  &  Evaluation  in  AFLC 


The  AFLC  conducts  continuing  assessment  of  the  reliability  of 
operational  systems  through  its  five  Air  Logistic  Centers  and  one 
Acquisition  Logistics  Division. 


•  San  Antonio  Air  Logistics  Center 

•  Oklahoma  City  Air  Logistics  Center 

•  Warner-Robins  (Georgia)  Air  Logistics  Center 

•  Sacramento  Air  Logistics  Center 

•  Ogden,  Utah  Air  Logistics  Center 

•  Air  Force  Acquisition  Logistics  Division 


V  v  v  V  v 
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The  focal  point  of  reliability  lor  the  five  centers  is  AFLC/LO  (Mr. 

Craig  Gridley,  513/257-3435,  WPAFB).  The  Acquisition  Logistics 

Division  of  AFLC  interfaces  with  acquisition  agencies  to  ensure  that 

a  more  supportable  system  is  developed.  This  division  contains  a 

reliability  group  whic-h  often  works  with  the  program  office  to  introduce 
the  logistic  point  of  view  early  in  the  development  cycle.  (The  AFALD 

focal  point  for  RAM  is  Mr.  William  Romas,  PTEA,  513/255-4028, 

WPAFB.  ) 

The  producibility,  reliability,  availability,  and  maintainability 
(PRAM)  program  office  at  Wright-Patterson  Air  Force  Base  is  res¬ 
ponsible  for  tracking  systems  after  initial  production  and  operational 
costs.  Each  of  the  five  Air  Logistics  Centers  have  a  PRAM  unit  to 
which  they  assign  their  own  personnel.  PRAM  uses  data  from  in-service 
testing,  but  does  not  design  reliability  assessment  tests. 

3.  Individuals  Contacted 

•  Maj.  Ned  Criscimanga,  HQ  USAF,  LGYE,  Engineering  and 
Support  Division,  202/695-0080 

•  Mr.  Elmer  Peterson,  HQUSAF  RDXM,  Office  of  Planning 
and  Program  Analysis,  202/697-6093 

•  Mr.  I.  N.  Shimi,  Directorate  of  Math  and  Information  Sciences, 

Air  Force  Office  of  Scientific  Research,  202/767-4939 

•  Capt.  Herbert  LaFlame,  PRAM  Element  Manager,  HQAF, 

RDPV,  202/697-5414 

•  Mr.  Tony  Athens,  Air  Force  Logistics  Command,  San  Antonio 
Air  Logistics  Center,  512/925-8961 

J 
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•  Col.  John  Hager,  Scientific  Advisory  Board,  202/697-8845 

•  Mr.  Anthony  Coppola,  RADC/RBRT,  315/330-4726 

•  Mr.  Jerome  Klion,  RADC/RBRT,  315/330-4726 

•  Mr.  Tony  Pettinato,  RADC,  Engineering  Support  Electronics 
Division,  315/330-4726 

•  Mr.  Marion  Williams,  Chief  Technical  Advisor,  Analysis 
Directorate,  AFTEC/OA,  505/264-3316 

•  Maj.  Guy  Morgan,  Reliability  and  Maintainability  Staff,  AFSC, 
SDDE,  301/981-3316 

•  Col.  Glen  O'Banion,  AFTEC,  Director  of  Logistics,  505/264-0321 

•  Mr.  David  Barber,  RADC,  315/330-4726 

•  Col.  Ben  Swett,  ODDR  &  E(T& E),  202/697-1130 

•  Mr.  Jan  Howell,  AFFTC/DOEES,  213/350-3066 

•  Capt.  Mahlon  H.  Long,  ADTC/SDEP,  305/872-3674 

•  Mr.  Charles  Burneka,  ASD/ENES,  216/478-4913 

•  Mr.  Frank  Van  Horn,  ESD/DRT,  207/478-4913 

•  Lt.  Col.  Kenneth  Blakney,  SAMSO/AWSR,  213/833-1182 


C.  U.  S.  NAVY 


All  reliability  test  planning  and  assessment  activities  are  conducted 
within  the  Naval  Materiel  Command  in  the  Navy  with  the  exception  of  those 
conducted  by  the  independent  test  and  evaluation  agency.  Comprehensive 
Operational  Test  and  Evaluation  Force  (COMOPTEVFOR),  which  reports 
to  the  Director  of  Research,  Development,  Training  and  Evaluation.  The 


y 


organizational  structure  of  the  Navy  is  depicted  in  Exhibit  4.  Within  the 
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NAVY  ORGANIZATIONS  INVOLVED  IN  RELIABILITY  ASSESSMENT 
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Materiel  Command  there  are  five  systems  commands.  Three,  namely 
NAVELEX,  NAVAIR,  and  NAVSEA,  have  their  own  logistic  support  organ¬ 
izations,  test  facilities,  development  programs,  and  reliability  expertise. 
Overall  reliability  policy  for  the  Materiel  Command  is  set  by  the  Deputy 
Chief  of  Naval  Materiel,  Reliability  and  Engineering. 

The  development  function  in  the  three  systems  commands  is  generally 
carried  out  by  a  designated  program  or  project  manager  who  takes  the 
project  from  the  initial  need  determination  through  technical  and 
operational  evaluation.  When  these  last  evaluations  are  satisfied, 
production  is  given  the  go-ahead  and  the  program  manager's  office  is 
usually  disbanded.  Designated  program  managers  report  directly  to  the 
Commander  of  each  systems  command.  Other  development  activities,  which 
occur  prior  to  designation  of  project  office  or  are  small  scale  efforts,  may 
often  be  found  within  specific  existing  offices. 

Testing  and  reliability  assessment  go  on  throughout  the  development 
cycle.  The  project  manager  either  hires  needed  experts  to  work  in  his 
office  or  draws  on  the  reliability  assessment  expertise  that  exists  within 
his  command,  especially  the  expertise  that  is  pertinent  to  the  equipment 
being  developed  (sonar,  power  plants,  etc.  ). 

Reliability  assessment  activities  peculiar  to  each  of  the  three  systems 
commands  are  discussed  in  detail  below. 
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1.  NAVAL  SEA  SYSTEMS  COMMAND  (NAVSEA) 

NAVSEA  contains  the  old  Naval  Ordnance  Systems  Command  and  the 
Naval  Ship  Systems  Command.  Its  directorates  which  are  directly  or  in¬ 


directly  involved  with  reliability  assessment  are  as  follows: 

•  Nuclear  Power  Directorate  (08) 

•  Weapons  Systems  and  Engineering  Directorate  (06) 

•  Fleet  Support  Directorate  (04) 

•  Research  and  Technology  Directorate  (03) 

•  Reliability,  Maintainability,  and  Quality  Assurance  Directorate 
(RM&QA)  (98) 

The  first  two  listed  are  responsible  for  program  development  activities. 
The  third,  the  Fleet  Support  Directorate,  is  concerned  with  logistical  sup¬ 
port.  The  Research  and  Technology  Directorate  is  responsible  for  the 
development  of  new  systems  prior  to  approval  for  full- scale  development. 
Finally,  the  RM&OA  Directorate  oversees  the  reliability,  maintainability, 
and  quality  assurance  activities  of  the  program  office, 
a.  Development  and  Acquisition 


Development  and  acquisition  activities,  other  than  for  Nuclear  Propul¬ 
sion  Systems  and  Ships,  are  focused  within  the  Weapons  Systems  and 
Engineering  Directorate  (06).  This  directorate  is  responsible  for  field 
activities  as  well  as  life  cycle  development  and  management,  fire  controls, 
sonar,  torpedoes,  and  other  weapon  related  systems.  Development  activities 


can  actually  begin  within  the  Research  and  Technology  Directorate  prior  to 
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designation  of  a  program  or  project  managers  (PM's)  in  06.  Also,  there 
are  project  offices  within  the  Fleet  Support  Directorate  and  some,  such  as 
the  Polaris  Office,  which  report  directly  to  the  Chief  of  Naval  Materiel. 

During  development,  PM's  call  on  the  existing  expertise  within  the 
various  directorates  and  offices  of  NAVSEA,  and,  in  particular,  within  06. 
The  three  reliability  assessment  contacts  within  06  are: 

•  Mr.  Anthony  Frizalone,  Surface  Weapons  Division,  202/692-1422. 

•  Mr.  John  Fleischman,  Underwater  Weapons  Division,  202/692-7896. 

•  Mr.  Toshio  Oishi,  NAVSEC,  202/692-6423. 

These  three  offices  help  the  PM's  in  establishing  test  plans  for  preproduc¬ 
tion  tests,  environment  qualification  tests,  and  R&M  demonstration  tests 
and  assessment.  They  also  act  as  technical  consultants  for  other  aspects 
of  reliability  programs, 
b.  Testing  and  Evaluation 

All  NAVSEA  testing  is  coordinated  through  the  Test  and  Evaluation 
Office  of  the  Weapons  System  and  Engineering  Directorate.  This  office, 
currently  under  the  direction  of  Captain  Horowitz  (06N),  provides  overall 
NAVSEA  coordination  with  COMOPTEVFOR.  In  addition,  the  RM&QA 
Directorate  monitors  all  reliability  assessment  activities,  reviews  test 
plans  for  timeliness  and  cost  effectiveness,  participates  in  design  reviews, 
and  coordinates  all  approval  recommendations.  The  Directorate  also  follows 


the  system  into  operation  to  see  if  it  attains  its  R&M  goals  in  the  fleet. 
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The  Test  and  Evaluation  Office  also  supports  Weapons  Quality  Engin-  1 
eering  Centers  (WQEC's)  at  Yorktown,  Concord,  Seal  Beach,  Indian  Head, 
Keyport,  Corona,  and  Crane,  Indiana.  The  WQEC's  conduct  reliability 
testing  on  assigned  products,  mostly  expendable  weapons  munitions.  They 
are  under  the  direction  of  Captain  Horowitz  (06N)  in  the  Test  and  Evaluation 
Office  (202/692-8212)  which  provides  overall  NAVSEA  coordination  with 
COMOPTEVFOR. 

The  Test  and  Evaluation  Office  coordinates  with  the  following: 

•  Fleet  Analysis  Center  (FLEETAC),  Corona,  California  -  Analyzes 
RM&A  data  from  the  fleet.  Mr.  Howard  Clark,  714/736-4211. 

•  Concord  Test  Center,  Concord,  California  -  Tests  munitions 
(bullets).  Mr.  Lawrence  Nichols,  415/671-2219. 

•  Seal  Beach .  Reliability  tests  on  ASROC,  SUBROC,  Marine  Corps 
Munitions  and  Missiles  and  all  TAC  and  STRATEGIC  Nuclear 
Weapons.  Integrate  lab  data  and  do  joint  service  and  DOE  Nuclear 
safety  testing.  Mr.  Lawrence  Grey,  213/596-9489. 

•  Navaj  Torpedo  Station,  Keyport  Washington,  Mr.  Charles  Thorn, 
206/396 -227 1 . 

•  Naval  Weapons  Station,  Yorktown,  Virginia  -  Tests  of  underwater 
mines  and  demolitions.  Jeff  Lamb,  804/887-4886. 

•  Indian  Head,  Indian  Head,  Maryland.  Mr.  John  Henderson, 
301/743-4324. 

All  ranges,  such  as  Point  Mugu  and  AWFTF,  are  managed  by  NAVAIR.  The 
tests  at  the  above  facilities  are  mostly  of  expendables. 

NAVSEA  does  not  do  much  in-service  testing.  Instead,  they  use  3M 
data  and  extensive  simulation  modeling.  Reliability  assessment  follows  a 
similar  pattern  in  NAVAIR  and  NAVELEX.  I 
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2.  NAVAL  AIR  SYSTEMS  COMMAND  (NAVAIR) 
a.  Development  Acquisition 

Acquisition  and  development  are  coordinated  by  24  program  manage  - 
(PMA's)  within  the  Directorate  for  Plans  and  Programs  (01).  As  in  NAV- 
SEA,  although  the  program  managers'  offices  are  usually  staffed  to  handle 
reliability  planning,  they  often  turn  to  other  NAVAIR  agencies  for  technical 
assistance  in  areas  such  as  avionics,  power  plants,  and  reliability  assess¬ 
ment. 

Other  directorates  involved  in  or  concerned  with  reliability  are: 

•  Test  and  Evaluation  (06) 

•  Air  Logisitics  (04) 

•  Weapons  Systems  (05) 

•  Research  and  Technology  (03) 

As  noted  above,  program  managers  are  within  AIR  01  but  some  pro¬ 
duct  managers  are  within  AIR  05.  Also  within  AIR  05  is  a  Relability  and 
Maintainability  Groups  (AIR  5205),  headed  by  Mr.  James  Wiggins,  202/ 
692-7595.  This  group  works  with  the  PMA's  by: 

•  analyzing  failure  data; 

•  establishing  reliability  criteria; 

•  reviewing  designs; 

•  investigating  failures;  and 

•  assisting  in  developing  reliability  program  plans. 
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NAVAIR  has  a  special  reliability  and  maintainability  group  in  03  which 
develops  new  R&M  applications  and  methods  and  develops  new  methods 
of  applying  R&M  to  military  systems.  For  example,  in  one  project  the 
group  is  exploring  R&M  procedures  to  use  early  in  the  development 
cycle.  Contacts  within  this  group  are: 

•  Mr.  Steven  Hurst,  Assistant  Technical  Administrator  for 
Logistics,  Hydraulics,  Mechanical  Equipment  and  Fluidics. 

•  Mr.  Fred  Hall,  Administrator,  Reliability  and  Maintainability 
for  Advanced  Technology  Programs  (3406). 

Both  can  be  reached  at  202/692-7443.  Mr.  Hall  is  also  NAVAIR’s  repre¬ 
sentative  on  the  inter- services  task  force  for  improved  R&M. 
b.  Testing  and  Evaluation 

Testing  is  coordinated  by  the  Test  and  Evaluation  Directorate  (AIR  06), 
whose  functions  include: 


•  manage  and  modernize  T&E  Bases  including  all  munitions  test 
facilities  used  by  NAVSEA  and  NAVALEX  as  well  as  NAVAIR: 

•  assist  NAVAIR  Program  Managers  in  structuring  the  best  T&E 
plan,  set  in  with  Program  Office  staff,  work  with  contractors, 
ensure  T&E  plan  is  timely  and  cost-effective  and  that  it  complies 
with  policy,  and  with  acquisition  plans  in  DOD  Directorates  5000.  1, 
.  2  and  .  3. 


Detailed  test  contents  are  worked  out  by  the  engineers  in  AIR  05. 

Currently,  emphasis  is  being  placed  on  (1)  using  simulations  to 
assess  reliability  in  the  Navy  since  test  resources  are  scarce,  systems 
more  costly  than  ever,  and  RAM  personnel  bogged  d.  wn  in  current  prob¬ 
lems;  and  (2)  following  up  failures  and  fixes  in  the  R&D  cycle  to  know  what 
to  look  for  in  operational  and  acceptance  testing.  Interest  in  the  Simula 
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tion  approach  stems  from  the  DOD  level  where  a  survey  on  simulation  is 
currently  being  administered  by  Air  Munitions  Requirements  and  Develop¬ 
ment  Command. 


3.  NAVAL  ELECTRONICS  COMMAND  (NAVELEX) 
a.  Development  and  Acquisition 

Development  and  acquisition  of  major  systems  are  coordinated  by 
seven  Project  Managers  (PME's)  who  report  directly  to  the  Chief  of 
NAVELEX.  Acquisition  of  other  items  and  products  is  coordinated  by 
Acquisition  Managers.  The  directorates  involved  in  reliability  assessment 
are: 

•  Logistics  (04) 

•  Research  and  Technology  (03) 

•  Material  Acquisition  (05). 

Reliability  test  plans  are  developed  by  the  PME's  and  acquisition  managers 
with  the  OPR  within  04:  Mr.  William  Wallace  (4702),  Reliability  Branch 
Head,  202/692-7526.  This  office  has  sign-off  authority  on  all  test  plans 
and  also  performs  a  monitoring  function  with  the  PME's  similar  to  Code 
98  in  NAVSEA.  Test  and  evaluation  coordination  is  handled  by  Capt.  L.  A. 
Dwyer  (05E)  whose  office  sets  up  the  T&E  Master  Plans  with  the  PME's 
and  coordinates  with  COMOPTEVFOR  for  completion  to  OPEVAL. 

NAVELEX  has  one  test  facility  at  Saint  Indigoes,  Maryland- -Naval  Elec¬ 
tronics  Systems  Test  and  Evaluation  Detachment  (NESTED). 

I _ _ _ J 
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4. 


INDIVIDUALS  CONTACTED  IN  ALL  COMMANDS 


NAVSEA 


•  Mr.  Howard  Fleck  (98),  Deputy  Commander  for  Reliability,  Main¬ 
tainability  and  Quality  Assurance,  202/692-3387. 

•  Mr.  Anthony  Frizalone  (06- G2),  Division  Director,  Reliability  and 
Quality  Engineering  (Surface  Weapons),  203/692-1426. 

•  Mr.  Henry  Itkin  (06-G2),  Assistant  for  Reliability  Statistical 
Analysis,  703/692-1426. 

•  Mr.  John  Fleischman  (06-H5),  Director,  Assurance  Engineering 
Office  (Underwater  Weapons),  202/692-7896. 

•  Mr.  Steven  Robling  (06-N1),  Director  of  Systems  Evaluation  Division, 
Test  and  Evaluation  Trials  and  Readiness  Office,  202/692-8212. 

•  Mr.  Melvin  Landis  (9821),  Reliability  and  Maintainability  Engineer, 
702/692-0415. 

•  Mr.  Toshio  Oishi  (61-81B),  Chief  Effectiveness  Section  Naval 
Systems  Effectiveness  Command,  202/692-6423. 

•  Ms.  Beatrice  Orleans,  Chief  Statistician,  202/692-9514. 

•  Mr.  Morton  Buckberg  (61-12),  Reliability  Section  Head,  Naval 
Ship  Engineering,  202/692-2150. 

•  Mr.  Donald  Johnson  (04- C) ,  Technical  Director ,  202/692-3526. 

NAVAIR 


•  Mr.  James  R.  Wiggins  (520  5),  Branch  Head  Reliability  and  Main¬ 
tainability,  202/692-7595. 

•  Mr.  Fay  Norton  (52051),  Reliability  Engineer,  202/692-7595. 

•  Mr.  Fred  Hall  (340G),  Administrator,  Reliability  and  Maintainability, 
202/692-7443. 

NAVE LEX 


•  Mr.  William  Wallace  (4702),  Reliability  Branch  Head,  202/692-7526. 
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•  Capt.  G.  D.  Webber  (348CP5),  Deputy  Assistant,  Reliability  and 
Engineering,  202/692-1106. 

•  Mr.  Bruce  McDonald  (436),  Statistician,  Office  of  Naval  Research, 
202/692-4315. 

•  Mr.  Ken  LaSalla,  Assistant  Director,  Program  Assessment  Divi¬ 
sion,  Reliability  and  Engineering,  202/692-1748. 

D.  MARINE  CORPS 

Most  of  the  Marine  Corps  systems  and  equipment  are  procured  in 
conjunction  with  Army  or  Navy  procurements.  In  these  cases,  the  Marine 
Corps  will  act  in  a  review  capacity  to  RAM  assessment  and  other  develop¬ 
ment  issues.  However,  the  Marines  do  their  own  testing  and  reliability 
assessment  for  amphibious  vehicles. 

As  in  the  other  services,  development  efforts  are  managed  by  an 
Acquisition  Program  Officer.  Two  managers,  Mr.  John  J.  Durant  (202/ 
694-2306),  and  Mr.  Gilbert  T.  Lussier  (202/694-2306),  handle  33  develop¬ 
ment  programs  without  special  assistance  in  any  of  the  technical  areas, 
such  as  finance,  quality  assurance,  training,  or  logistics.  They  are 
responsible  for  four  primary  functions: 

•  value  engineering; 

•  quality  assurance; 

•  configuration  management;  and 

•  reliability,  availability,  and  maintainability. 

The  independent  test  activity  is  conducted  at  the  Marine  Corps  Development 
and  Education  Center  at  Ouantico  by  the  Marine  Corps  Tactical  Systems 
Support  Activity  (MCTSSA).  The  MCTSSA  contact  for  amphibious  vehicles 


is  Mr.  John  Carr,  703/640-2242. 
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E.  TEST  FACILITIES  IN  THE  DEPARTMENT  OF  DEFENSE 


The  test  facilities  in  DOD  can  be  divided  into  two  groups  as  follows: 
1.  NATIONAL  RANGES 


Those  major  DOD  ranges  and  test  facilities  which  are  unique  national 


assets  designed  to  support  requirements  of  major  DOD  programs  and 


other  Federal  Government  Agencies. 


National  Ranges 


Management  Agency 


White  Sands  Missile  Range  (WSMR) 

Kwajalein  Missile  Range  (KMR) 

Pacific  Missile  Range  (PMR) 

National  Parachute  Test  Range  (NPTR) 

Eastern  Test  Range  (ETR) 

Space  &  Missile  Test  Center  (SAMTEC) 

Satellite  Control  Facility  (SCF) 

Arnold  Engineering  Development  Center  (AEDC) 


U.  S.  Army 
U.  S.  Army 
U.S.  Navy 
U .  S.  Navy 
U.  S.  Air  F orce 
U.  S.  Air  Force 
U.  S.  Air  F orce 
U.  S.  Air  Force 


2.  DOD  MAJOR  TEST  FACILITIES 


Those  other  major  DOD  test  facilities  which  support,  almost  en¬ 
tirely,  DOD  requirements: 


Test  F acilities 


Management  Agency 


Dugway  Proving  Ground  (DPG) 

Arctic  Test  Center  (ATC) 

Tropic  Test  Center  (TTC) 

Yuma  Proving  Ground  (YPG) 

Jefferson  Proving  Ground  (JPG) 

Electronic  Proving  Ground  (EPG) 

Aberdeen  Proving  Ground  (APG) 

(Materiel  Test  Directorate  Only) 

Atlantic  Underwater  Test  & 

Evaluation  Center  (AUTEC) 

Naval  Air  Test  Center  (NATO 

Naval  Air  Propulsion  Test  Center  (NAPTC) 

Naval  Air  Test  Facility  (NATF) 


U.  S.  Army 
U.S.  Army 
U.  S.  Army 
U.  S.  Army 
U.S.  Army 
U.  S.  Army 
U .  S.  Army 


U.  S. 
U.S. 
U.  S. 
U.S. 


Navy 

Navy 

Navy 

Navy 


J 


MAXIMUS 


Management  Agency 


Test  Facilities 


Naval  Weapons  Center  (NWC) 

(T&E  Portion  Only) 

Atlantic  Fleet  Weapons  Range  (AFWR) 

Air  Force  Special  Weapons  Center 
(Incl.  6585th  Test  Group)  (AFSWC) 

Tactical  Fighter  Weapons  Center 
(TFWC)  (Continental  Operations  Range  Only) 
Air  Force  Flight  Test  Center  (AFFTC) 
Armament  Development  and  Test  Center 
Air  Defense  Weapons  Center  (ADWC) 


U.  S.  Navy 

U.  S.  Navy 
U.  S.  Air  F orce 

U.  S.  Air  Force 

U.  S.  Air  Force 
U.  S.  Air  F orce 
U.  S.  Air  F orce 


II.  PURPOSE  AND  SCOPE  OF  SURVEY 


This  section  presents  the  results  of  a  brief  survey  of  the  current 
reliability  assessment  programs  in  the  Army,  Air  Force,  Navy,  and 
Marine  Corps.  The  categories  of  information  sought  were: 

•  organizations  involved  in  reliability  assessment; 

•  resources  expended  on  testing; 

•  potential  for  reducing  test  costs  by  using  prior  information 
on  component  reliability; 

•  obstacles  or  barriers  to  the  utilization  of  Bayesian  techniques; 

•  amenability  to  demonstration  of  "A  Bayesian  Scheme  for 
Sequentially  Testing  a  Multi- Component  System.  " 

Approximately  50  interviews  were  conducted  by  telephone  or  in 
person  using  an  interview  guide  shown  in  Section  V.  Among  the  inter¬ 
viewees  were  persons  in  coordinating  roles,  research  scientists, 
statisticians,  and  program  development  managers.  The  variation  in 
roles,  perspectives,  and  backgrounds  of  the  interviewees  made  standardi 
zation  of  the  questions  - -and,  hence,  statistical  analysis  of  responses-- 
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impractical.  The  analysis  of  survey  results,  thus,  is  of  a  general, 
descriptive  nature. 

The  survey  was  conducted  primarily  by  telephone  using  the  interview 
guide  cited  earlier.  Respondents  were  selected  from  referrals  made  by 
other  respondents  or  by  their  supervisors.  There  was  no  attempt  made  to 
randomize  selection  of  respondents;  thus,  from  a  statistical  standpoint 
the  results  cannot  be  said  conclusively  to  represent  the  views  held  by 
non-respondents.  However,  the  consistency  of  answers  provided  makes 
this  "problem"  in  the  survey  technique  immaterial. 

In  length,  the  interviews  varied  from  5  to  20  minutes.  The  interview¬ 
er  described  the  purpose  of  the  survey  as  "to  explore  the  amenability  of 
various  programs  within  the  Armed  Forces  to  adopt  the  use  of  Bayesian 
techniques  in  the  reliability  assessment  process.  "  To  accomplish  this  end, 
the  interviewee  was  told  that  we  were  seeking  information  regarding: 

•  the  types  of  testing  being  conducted  within  his  organization; 

•  the  resources  expended  on  testing; 

•  the  obstacles  that  existed  to  introducing  Bayesian  techniques;  and 

•  the  potential  test  sites  where  the  subject  technique  could  be  demon¬ 
strated. 

A  general  discussion  usually  ensued  during  which  the  interviewer 
recorded  the  relevant  information.  When  this  discussion  ended,  the 
interviewer  asked  for  answers  to  questions  on  the  interview  guide  which 
had  not  been  answered  during  the  general  discussion.  The  results  of  this 
process  are  presented  in  the  following  section. 
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III.  FINDINGS  OF  SURVEY 

In  this  section,  we  present  a  summary  of  the  information  gathered 
from  the  survey  in  sections  corresponding  to  the  subject  areas  of 
the  interview. 

A.  TYPES  OF  TESTING 

Generally  speaking,  reliability  tests  are  performed  during  research 
and  development  phases,  prior  to  and  during  initial  production,  and  after 
storage  or  limited  use.  During  development,  tests  are  usually  conducted 
by  a  contractor  under  a  test  plan  agreed  to  by  the  government.  As  the  items 
are  developed  to  the  pre-production  phase,  the  Service  typically  takes  more 
testing  responsibility  to  ensure  compliance  under  operational  conditions. 

Our  questions  centered  around  testing  complete  systems.  In  the  main, 
these  questions  did  not  elicit  a  set  of  responses  that  could  be  generalized. 
Most  persons  queried  could  not  really  provide  a  description  of  their  program 
in  terms  of  developmental,  production,  or  operational  testing.  Variations 
in  terminology  also  masked  the  results.  For  example,  field  failures  of 
equipment  have  made  program  managers  keenly  aware  of  the  need  to  test 
equipment  under  operational  or  simulated  operational  conditions.  Hence, 
operational  tests  do  not  always  refer  to  tests  of  a  system  that  has  already 
been  in  use. 

The  interviews  revealed  that  there  is  only  a  limited  amount  of  relia¬ 
bility  assessment  accomplished  for  major  assembled  systems  for  several 
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•  the  MTBF's  are  too  long  to  develop  accurate  estimates; 

•  the  systems  are  too  expensive  to  test  to  destruction; 

•  the  reliability  levels  needed  are  thought  to  be  too  high  to  be 
demonstrated  statistically; 

•  there  is  less  interest  in  the  point  estimate  of  reliability 
and  its  confidence  interval  than  in  other  facets  of  the 
system  such  as  operability,  ease  of  handling,  and 
specific  failure  modes. 

B.  RESOURCES  EXPENDED 

Most  of  those  interviewed  were  not  aware  of  any  efforts  to  collect 
or  analyze  data  on  the  resources  expended  in  reliability  assessment, 
except  with  respect  to  individual  program  development  efforts  where 
the  amount  expended  is  contractually  regulated.  Even  under  contractual 
development  efforts,  however,  there  does  not  appear  to  be  any  guidance 
developed  from  past  experience  on  how  much  should  be  spent  on  reliability 
assessment  or  how  reliability  assessment  costs  should  be  controlled. 

Some  interviewees  commented  that  reliability  assessment  costs 
are  not  as  easy  to  define  as  maintenance  costs.  Maintenance  is  ongoing 
and  more  readily  costable.  Other  comments  pointed  to  the  difficulty 
of  distinguishing  between  some  engineering  and  equipment  costs  and  relia¬ 
bility  assessment  costs.  In  a  few  cases,  interviewees  indicated  that 
they  could  not  answer  questions  on  cost  unless  we  could  substantiate 
our  "need  to  know.  " 

One  interviewee  had  just  completed  a  cost  study  of  20  systems,  all 
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of  which  were  developed  under  contract.  His  organization  used  a 
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questionnaire  approach  followed  by  interviews  in  order  to  standardize 
answers.  The  study  resulted  in  empirical  relationships  between 
reliability  costs  and  other  variables  such  as  type  of  contract,  number 
of  active  parts,  and  number  of  deliverables.  Copies  of  this  report 
have  not  been  made  available  as  yet.  Information  can  be  obtained  from: 
NAVSEA  06-G2,  Washington,  D.  C.  20362. 

C.  BAYESIAN  TECHNIQUES 

Most  of  the  persons  interviewed  who  were  familiar  with  Bayesian 
techniques,  expressed  some  frustration  at  the  problems  or  obstacles 
that  prevented  their  implementation.  For  example,  manufacturers  may 
be  loathe  to  try  to  work  with  new  reliability  acceptance  standards  when 
they  are  sure  of  a  profit  from  the  old  ones.  Moreover,  Bayesian  tech¬ 
niques  are  still  strongly  opposed  by  "classical"  statisticians  and,  thus, 
are  controversial.  However,  even  among  those  interviewees  who  held 
to  the  classical  view,  there  was  an  interest  in  seeing  demonstrations 
conducted  using  Bayesian  approaches. 

Responses  to  questions  concerning  the  barriers  to  the  imple¬ 
mentation  of  Bayesian  techniques  were  possible  to  tabulate,  as 
shown  in  Exhibit  5  on  the  following  page.  Some  comments  on  the 
validity  of  the  barriers  were  voiced.  For  example,  a  few  inter¬ 
viewees  felt  that  data  applicability  was  not  a  serious  limitation. 

They  felt  that,  although  it  was  true  that  field  environments 
differ  from  test  environments,  the  cost  of  testing  systems  in  the 
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Exhibit  5 

OBSTACLES  TO  THE  IMPLEMENTATION  OF 
BAYESIAN  TECHNIQUES 


Obstacle 


1.  Selection  of  prior  is  problematic 

2.  Data  shortages  and  cost  of  retrieval  are  high 

3.  Component  data  are  not  applicable  because  of 
difference  between  test  stand  and  operational 
conditions 

4.  Problem  of  acceptance  of  subjective  judgment 


Number  of 
Mentions 

5 

5 

5 

4 


5.  Contractual  difficulties 


2 
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field  is  so  high  that  one  is  forced  to  use  whatever  relevant  data  exists. 

With  respect  to  familiarity  with  Bayesian  techniques,  the  answers 
were  difficult  to  quantify  because  of  the  irregularity  and  overlaps  in  the 
nominal  scales  used.  What  can  be  said  is  that  those  thought  to  be  familiar 
were  only  partially  so.  What  constituted  "familiarity"  may  have  varied 
among  interviewees. 

All  of  the  Services  sponsor  training  programs  in  the  acquisition 
process  and  in  reliability  techniques.  With  a  few  exceptions,  such  as 
the  AFIT  Master's  Degree  program,  the  courses  are  basic,  and 
Bayesian  techniques  are  not  treated  in  any  depth.  The  consensus  seems 
to  be,  however,  that  there  are  knowledgeable  persons  throughout  the 
research-oriented  commands,  at  the  Rome  Air  Development  Center,  and 
the  Office  of  Naval  Research,  but  a  general  void  at  the  program  monitor¬ 
ing  and  field  testing  levels.  Most  agreed  that  demonstration  projects 
using  Bayesian  techniques  either  would  be  helpful  or  were  critically 
needed.  According  to  an  interviewee  in  a  coordinating  role  in  the 
Army's  reliability  assessment  network,  "in  certain  areas  we  are  being 
forced  into  the  use  of  Bayesian  techniques,  especially  with  one-shot 
devices  which  must  be  destructively  tested.  " 

Most  interviewees  were  concerned.  Some  who  had  been  thinking  of  the 
problem  suggested  developing  computer  simulations  of  the  test  situation 
to  substantiate  the  savings  that  could  accrue.  Others  informed  us 
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of  the  one  or  two  applications  that  have  been  tried,  and  some  suggested 
the  types  of  systems  (generally  expensive  and  test-destroyable)  that 
would  be  most  suited  to  the  application. 

Several  people  reported  on  the  many  attempts  to  develop  Bayesian 
material  to  be  included  in  MIL  STD  781  that  have  gone  astray  at  the  last 
minute.  One  reliability  analyst  reported  on  work  conducted  by  a  noted 
scholar  describing  Bayesian  procedures  for  inclusion  in  MIL  STD  781. 

The  recommended  procedures  were  later  deemed  inappropriate  for  the 
standards  according  to  a  review  by  another  noted  scholar. 

Experience  has  shown  that  the  Government  must  be  protected  from 
the  choice  of  prior  distributions  which  may  bias  the  results  in  favor 
of  the  contractor's  claims.  This  lesson  was  learned  during  development 
of  SONAR  equipment  by  General  Electric.  Those  familiar  with  this 
situation  suggested  that  a  process  for  reaching  consensus  on  prior 
distributions  must  be  established. 

One  interviewee  reported  on  being  involved  for  three  to  four  years 
in  a  Tri-Service  group  which  attempted  to  develop  applications  of 
Bayesian  statistics.  He  was  not  aware  of  the  final  disposition  except 
that  some  of  the  methods  were  still  being  used  at  Picatinny  Arsenal. 

One  respondent  at  the  Rome  Air  Development  Center  reported  on  an 
extensive  publication,  RADC-TR-76-294.  This  report  does  not  deal 
with  cost-effective  testing,  but  it  does  deal  in  depth  with  Bayesian 


(and  related)  acceptance  plans. 
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D.  SYSTEM  AND  TEST  CHARACTERISTICS 

Two  questions  of  interest  ares  (1)  to  what  extent  do  the  systems 
tested  contain  redundancies,  and  (2)  to  what  extent  is  destructive  testing 
used.  Although  these  questions  were  addressed  only  selectively  (many 
interviewees  were  not  directly  involved  in  testing  a  particular  system), 
the  answers  are  summarized  in  Exhibit  6  on  the  following  page. 

The  question  on  "destructive  tests"  was  actually  too  simplistic. 
The  operationally  oriented  tests,  performed  by  agencies  such  as  AFTEC 
which  represent  the  users'  point  of  view,  try  to  establish  whether  the 
equipment  will  pass  or  fail  certain  mission  stresses.  In  such  cases, 
if  destruction  occurs,  it  is  not  a  planned  part  of  the  test.  The  fact  that 
the  survey  was  conducted  across  those  involved  in  development  as  well 
as  operational  testing  explains  some  of  the  variance  in  responses. 

E.  TECHNICAL  ISSUES 

A  few  technical  issues  were  brought  up  by  interviewees  which 
could  effect  implementation  of  the  method  being  studied.  The  first 
problem  concerns  the  fact  that  prior  test  data  on  components  may  not 
be  appropriate  unless  the  stress  environment  to  which  the  component 
was  subjected  duplicates  what  will  exist  in  the  system  operation. 

Also,  some  failures  may  occur  because  of  "interconnect"  problems; 
that  is,  the  components  function  but  the  mechanisms  connecting  them 
do  not.  While  these  certainly  ar»  problems,  partial  data,  used  judi¬ 
ciously,  are  better  than  n'  Jata  at  all.  Also,  the  number  of  systems 
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Exhibit  6 

SYSTEM  REDUNDANCY  AND  USE  CF  DESTRUCTIVE  TESTS 

Redundancies  Destructive  Tests 

Depends  on  the  system  3  Not  planned  4 

Some  have  a  lot  1  Occasionally  3 

A  lot  do  have  some  3  Frequently  2 

Not  many  1  Hardly  at  all  2 


Not  answered 


3 


Depends  on  system  1 
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tests  needed  to  discover  these  problems  is  surely  less  than  that 
needed  to  establish  reliability  estimates  for  the  entire  system. 

A  second  problem  expressed  by  some  interviewees  was  that 
systems  are  not  tested  on  a  component-by-component  basis.  They 
are  tested  as  systems,  but  it  is  ultimately  the  components  which  fail. 
Hence,  each  system  test  amounts  to  a  test  on  each  component.  What 
we  are  proposing  is  that  there  are  more  economical  ways  of  testing 
the  components  and  deriving  system  reliability  estimates  than  by 
complete  system  tests. 

In  sum,  a  great  deal  of  interest  was  sparked  by  the  notion  of  optimal 
or  cost-effective  testing  using  Bayesian  techniques.  Persons  in  all 
four  services  agreed  that  the  time  has  come  to  provide  a  systematic 
rationale  for  structuring  test  plans  which  considers  the  costs  of  testing 
and  the  value  of  the  information  obtained. 
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IV.  SURVEY  INSTRUMENT:  RELIABILITY  ASSESSMENT  QUESTIONNAIRE 


1. 

Type  of  testing 

Development 

% 

Production 

% 

Other  % 

2.  Testing  done  by 

Military  % 

Contractor  _% 

3.  Approximate  $  per  year  absorbed  by  reliability  assessment _ . 

4.  Who  is  usually  the  dominant  force  in  determining  the  nature  of  the  test 
plan? 

Program  Officer  _ _ 

Contractor  _ 

Other  _ 

5.  Familiarity  of  program  officers  with  Bayesian  techniques: 


Some 

Very 

Partially 

Half 

Very 

Partially 

Most 

Very 

Partially 

Hardly  Any 

Very 

_ Partially 

6.  Obstacles  or  barriers  to  the  implementation  of  Bayesian  techniques. 


J 
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7. 


Suggestions  for  overcoming  barriers. 


8. 


Applications  using  Bayesian  techniques  in  test  plan  development. 


9.  Methods  of  introducing  innovative  approaches  in  testing. 


10.  Demonstration  projects  using  Bayesian  techniques  for  the 
formulation  of  test  plans: 

Are  critically  needed  _ 

Are  sorely  needed  _ 

Could  be  helpful  _ 

Would  be  of  little  use  _ 
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11.  Rank  these  barriers: 

Program  Officer  knowledge  _ 

Contractual  problems  _ 

Selection  of  appropriate  prior  distribution  _ 

Lack  of  acceptance  of  Bayesian  techniques  _ 

12.  Does  your  command  do  reliability  testing  of  operational  systems? 


Are  these  system  tests  done  on  a  component-by- component  basis' 


Component-by- component 
System  Use  Test 
Other  (specify) 


14.  To  what  extent  do  systems  you  deal  with  contain  redundancies? 


To  what  extent  is  destructive  testing  used  in  your  command: 

Frequently  _ 

Occasionally  _ 

Hardly  at  all  _ 

Are  destructive  tests  performed  on  components  of  production  systems 


17. 


Does  your  command  maintain  any  cost  data  on  testing?  Exp] 


18. 


What  is  the  possibility  of  demonstrating  a  sequential 
scheme  in  your  command? 


Baye 
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DATE:  _ 

NAME:  _ 

ORGANIZATION: 
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THEN.  SOME  COMPONENTS  SHOULD  BE  TESTED 
MORE  THAN  OTHERS. 


BAYESIAN  APPROACH  TO  OPTIMAL  TESTING 
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A  BRA  M3 

PROGRAM  FLOW  CHART 


SUBROUTINES 

INDATA 

PRIORS 

SYSPRI 

SYSREL 

DECIDE 

UPDATE 

BETA 

MOMNT 


CONFIGURATION  CODE 
COMPONENT  ID 
COST  OF  TESTING 
PRIOR  KNOWLEDGE 
SYSTEM  TEST  RESULTS 


COMPUTE 

MOMENTS 

OF 

COMPONENT 

DISTRIBUTION 


COMPUTE 

MOMENTS 

C 

SYS 

DISTRJ 

>F 

STEM 

.BUTTON 

UPDATE 

MOMENTS 

OF 

COMPONENTS 
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/  IN 
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[PUT 

EST 

5ULTS 

PRINT 

RESULTS 

FOR 

CHECK 


COMPUTE 
EXPECTED  RISK 
FOR  EACH 
COMPONENT 


PRINT 

EXPECTED  RISK 

AND  COST  OF 

> 

V 

TESTINC^^* 

YES 
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INPUT  DATA  FOR  PROGRAM 


MAX  =  NUMBER  OF  CONFIGURATION  CODE  ENTRIES 
I  CM  AX  =  NUMBER  OF  DISTINCT  COMPONENTS 
IC(I)  =  CONFIGURATION  CODE  FOR  ENTRY  I. 

ICN(I)  =  COMPONENT  NUMBER  FOR  ENTRY  I 

ICOST(I)  =  COST  OF  TESTING  THE  COMPONENT  ENTRY  I 

NUMPRI  =  NUMBER  OF  PRIOR  DISTRIBUTIONS  TO  BE  SPECIFIED 

IA( J)  =  PSEUDO  SUCCESSES  FOR  JTH  COMPONENT 

IB( J)  =  PSEUDO  FAILURES  FOR  JTH  COMPONENT 


CARD  n 


CARD  #2 


CARD  #MAX+1 


CARD  #MAX+2 


CARD  #MAX+ 

NUMPRI 


MAXIMUS 
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M2A1  AND  M3  INCREMENT  HOLDERS 


0 


Q 


K 


0 


MIL-D-1A1A0C  (FA) 


A. A. 3  Testing. 


A. A. 3.1  Chcnlcal  sailing.  -  A  representative  saaple  shall 
ba  randomly  selected  for  the  chemical  taita  iptelfltd  In  A. 5. 
the  procedure  daserlbad  In  ASM  B300  should  ba  uaad  to  withdraw 
the  saaplea.  The  ehaaleal  tests  In  A. 5  shall  be  performed  using 
prescribed  analytical  proeadura  for  replicate  detaral nation 
given  In  standard  analytical  textbooks. 


A. A. 3.2  Moisture. 


•A. A. 3.2.1  Moisture  eontent  of  lead  aside  (see 
Ma.1or  A  defect.  -  The  aolsture  content  oft 
be  dete rained  for  each  lot  at  the  tlae  of  loading.  A  saaple 
In  sufficient  quantity  to  perform  the  teat  detailed  In  A.5.1 
shall  be  selected  and  tasted  as  specified  In  A.5.1.  If  the 
moisture  content  Is  In  excess  of  the  percentage  specified  and 
loading  has  not  beeim.  the  lot  of  lead  aside  shall  he  related 
If  loading  has  begun,  action  on  .the  unloaded  lead  aside  shall 
be  as  specified  above 


j  W* rr. i 4 iwrrrr tu rrrr 


nTi.i'Xr.nn 


ant  content  of  ROT  (see  dw 


r  A  defect.  -  The  binder-lubricant  content  o 

■-ST* for  each  lot  at  the  tine  of  loading.  A  sample  of 
5  grane  shall  be  selected  froa  each  lot  and  tested  as  specified 
in  A. 5. 2.  If  the  binder-lubricant  eontent  Is  In  excess  of  the 
peroentage  specified  and  If  loading  has  not  begun,  the  lot  of 
ROT  shall  be  rejected  isitll  proper  corrective  action  has  been 
aoeo*>llshed  as  verified  by  repeating  the  binder-lubricant 
determination  specified  In  A. 5.2.  If  loading  has  begun,  action 
on  the  tsiloaded  ROT  shall  be  as  specified  above  and  all  detonators 
loaded  with  the  questionable  RSI  shall  be  rejected. 


A. A. 3. A  Functioning 


or  A  defect 
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FACTORS  CONSIDERED  IN  ADAPTING  THE  METHODOLOGY 
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